You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2025/06/20 09:16:18 UTC
Python 3.13 is coming
Hey here,
Over the last few days and weeks I made significant progress with Python
3.13 support and while the PR is not yet fully green - we are getting
close. I undrafted my PR recently:
https://github.com/apache/airflow/pull/46891
I have a kind request for others to start reviewing it, I will be happy to
respond to questions and I am also happy to accept some fixups to solve
things in a better way if possible, I did a few hacks and left some TODOs
that might need some creative ideas to be solved in a better, simpler way.
I split the PR into several commits - and keep it continuously rebased on
top of the latest main. I had to remove some providers (including FAB -
which is actually cool, because we can finally see that airflow codebase is
finally working even without FAB being around) - some of them are yet to
implement Python 3.13 (for example Apache Beam is working on it) so we
might add them back soon.
Note - Python 3.13 was a WAY more difficult migration than Python 3.12 -
there were a number of things removed/changed and small behavioral changes
that we relied on, also the dependencies are catching up way later, and
there are some tricky dependencies of ours that make things more complex
when it comes to selecting "min" versions of those. But we are finally
getting there.
Looks like we will be on-time to have 3.13 support for Airflow 3.1. We
might also attempt to cherry-pick it for 3.0, but it might be too much of a
hassle, so we can decide after we merge this one to main.
J.
Re: [HELP NEEDED} Python 3.13 is coming - converting serve_logs to fast_api
Posted by Jarek Potiuk <ja...@potiuk.com>.
Wooohooo!! I was afraid I will have to vibe-code it :)
On Mon, Jun 30, 2025 at 5:33 PM Zhe You Liu <ja...@apache.org> wrote:
> Sure, I'm on it!
> I had add the "Refactor serve_logs with FastAPI" in my backlog for a few
> months ago but interrupt by other task, almost finish this.
>
> Best.
> Zhe-You Liu
>
> On 2025/06/30 10:20:49 Jarek Potiuk wrote:
> > Hello here,
> >
> > I've been iterating on Py 3.13 support and have it "almost" green.
> >
> > https://github.com/apache/airflow/pull/46891
> >
> > Currently (until connexion 2.15.0 release) - we cannot make it work with
> > FAB so FAB (Among others) is excluded.
> >
> > However this brings the need to fix one more incompatibility (except that
> > we need to wait for at least RC release of the next provider's wave) -
> > serve_logs implicitly depends on Flask/Fab.
> >
> > Currently our "serve logs" method to serve logs for celery workers is
> still
> > part of the "airflow-core" and indirectly depends on Flask/FAB APP being
> > installed.
> >
> > There was a discussion that it should be possible to easily rewrite it
> > using fast-api, and it's the time we actually need it to:
> >
> > * have Python 3.13 support
> > * become truly FAB-independent
> >
> > I created an issue for it https://github.com/apache/airflow/issues/52526
> -
> > and since I have not been involved in fast-api development, I guess it
> will
> > be faster by someone who was.
> >
> > Can someone help with it ?
> >
> > J.
> >
> >
> >
> >
> >
> > On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > I also started to do more preparation and extracting things that we
> will
> > > need to merge first and release providers with some limits lifted. We
> need
> > > to make our providers released in PyPI prepared upfront because
> currently
> > > we are not able to generate PyPI constraints for Python 3.13 - I
> already
> > > have one of those merged -
> https://github.com/apache/airflow/pull/51994
> > > where we have to exclude `ray` from Python 3.13 and release provider
> before
> > > we get the PR "green" - similarly I have just created
> > > https://github.com/apache/airflow/pull/52060 to all providers that are
> > > currently limiting pandas (unnecessary it seems from my tests in 3.13
> PR).
> > >
> > > J.
> > >
> > >
> > > On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <am...@gmail.com>
> > > wrote:
> > >
> > >> Thanks for taking this on Jarek.
> > >>
> > >> These are not easy tasks to work on and I cannot think of a better
> person
> > >> to do it than you!
> > >>
> > >> I will review it shortly this week.
> > >>
> > >> I think it is a good thing to introduce in Airflow 3.1 and I would
> not be
> > >> too much in favour
> > >> to cherry pick it to the 3.x series, just for convenience sake.
> > >>
> > >> Thanks & Regards,
> > >> Amogh Desai
> > >>
> > >>
> > >> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >>
> > >> > Hey here,
> > >> >
> > >> > Over the last few days and weeks I made significant progress with
> Python
> > >> > 3.13 support and while the PR is not yet fully green - we are
> getting
> > >> > close. I undrafted my PR recently:
> > >> >
> > >> > https://github.com/apache/airflow/pull/46891
> > >> >
> > >> > I have a kind request for others to start reviewing it, I will be
> happy
> > >> to
> > >> > respond to questions and I am also happy to accept some fixups to
> solve
> > >> > things in a better way if possible, I did a few hacks and left some
> > >> TODOs
> > >> > that might need some creative ideas to be solved in a better,
> simpler
> > >> way.
> > >> >
> > >> > I split the PR into several commits - and keep it continuously
> rebased
> > >> on
> > >> > top of the latest main. I had to remove some providers (including
> FAB -
> > >> > which is actually cool, because we can finally see that airflow
> > >> codebase is
> > >> > finally working even without FAB being around) - some of them are
> yet to
> > >> > implement Python 3.13 (for example Apache Beam is working on it) so
> we
> > >> > might add them back soon.
> > >> >
> > >> > Note - Python 3.13 was a WAY more difficult migration than Python
> 3.12 -
> > >> > there were a number of things removed/changed and small behavioral
> > >> changes
> > >> > that we relied on, also the dependencies are catching up way later,
> and
> > >> > there are some tricky dependencies of ours that make things more
> complex
> > >> > when it comes to selecting "min" versions of those. But we are
> finally
> > >> > getting there.
> > >> >
> > >> > Looks like we will be on-time to have 3.13 support for Airflow 3.1.
> We
> > >> > might also attempt to cherry-pick it for 3.0, but it might be too
> much
> > >> of a
> > >> > hassle, so we can decide after we merge this one to main.
> > >> >
> > >> > J.
> > >> >
> > >>
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [HELP NEEDED} Python 3.13 is coming - converting serve_logs to fast_api
Posted by Zhe You Liu <ja...@apache.org>.
Sure, I'm on it!
I had add the "Refactor serve_logs with FastAPI" in my backlog for a few months ago but interrupt by other task, almost finish this.
Best.
Zhe-You Liu
On 2025/06/30 10:20:49 Jarek Potiuk wrote:
> Hello here,
>
> I've been iterating on Py 3.13 support and have it "almost" green.
>
> https://github.com/apache/airflow/pull/46891
>
> Currently (until connexion 2.15.0 release) - we cannot make it work with
> FAB so FAB (Among others) is excluded.
>
> However this brings the need to fix one more incompatibility (except that
> we need to wait for at least RC release of the next provider's wave) -
> serve_logs implicitly depends on Flask/Fab.
>
> Currently our "serve logs" method to serve logs for celery workers is still
> part of the "airflow-core" and indirectly depends on Flask/FAB APP being
> installed.
>
> There was a discussion that it should be possible to easily rewrite it
> using fast-api, and it's the time we actually need it to:
>
> * have Python 3.13 support
> * become truly FAB-independent
>
> I created an issue for it https://github.com/apache/airflow/issues/52526 -
> and since I have not been involved in fast-api development, I guess it will
> be faster by someone who was.
>
> Can someone help with it ?
>
> J.
>
>
>
>
>
> On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > I also started to do more preparation and extracting things that we will
> > need to merge first and release providers with some limits lifted. We need
> > to make our providers released in PyPI prepared upfront because currently
> > we are not able to generate PyPI constraints for Python 3.13 - I already
> > have one of those merged - https://github.com/apache/airflow/pull/51994
> > where we have to exclude `ray` from Python 3.13 and release provider before
> > we get the PR "green" - similarly I have just created
> > https://github.com/apache/airflow/pull/52060 to all providers that are
> > currently limiting pandas (unnecessary it seems from my tests in 3.13 PR).
> >
> > J.
> >
> >
> > On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <am...@gmail.com>
> > wrote:
> >
> >> Thanks for taking this on Jarek.
> >>
> >> These are not easy tasks to work on and I cannot think of a better person
> >> to do it than you!
> >>
> >> I will review it shortly this week.
> >>
> >> I think it is a good thing to introduce in Airflow 3.1 and I would not be
> >> too much in favour
> >> to cherry pick it to the 3.x series, just for convenience sake.
> >>
> >> Thanks & Regards,
> >> Amogh Desai
> >>
> >>
> >> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >> > Hey here,
> >> >
> >> > Over the last few days and weeks I made significant progress with Python
> >> > 3.13 support and while the PR is not yet fully green - we are getting
> >> > close. I undrafted my PR recently:
> >> >
> >> > https://github.com/apache/airflow/pull/46891
> >> >
> >> > I have a kind request for others to start reviewing it, I will be happy
> >> to
> >> > respond to questions and I am also happy to accept some fixups to solve
> >> > things in a better way if possible, I did a few hacks and left some
> >> TODOs
> >> > that might need some creative ideas to be solved in a better, simpler
> >> way.
> >> >
> >> > I split the PR into several commits - and keep it continuously rebased
> >> on
> >> > top of the latest main. I had to remove some providers (including FAB -
> >> > which is actually cool, because we can finally see that airflow
> >> codebase is
> >> > finally working even without FAB being around) - some of them are yet to
> >> > implement Python 3.13 (for example Apache Beam is working on it) so we
> >> > might add them back soon.
> >> >
> >> > Note - Python 3.13 was a WAY more difficult migration than Python 3.12 -
> >> > there were a number of things removed/changed and small behavioral
> >> changes
> >> > that we relied on, also the dependencies are catching up way later, and
> >> > there are some tricky dependencies of ours that make things more complex
> >> > when it comes to selecting "min" versions of those. But we are finally
> >> > getting there.
> >> >
> >> > Looks like we will be on-time to have 3.13 support for Airflow 3.1. We
> >> > might also attempt to cherry-pick it for 3.0, but it might be too much
> >> of a
> >> > hassle, so we can decide after we merge this one to main.
> >> >
> >> > J.
> >> >
> >>
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
[HELP NEEDED} Python 3.13 is coming - converting serve_logs to fast_api
Posted by Jarek Potiuk <ja...@potiuk.com>.
Hello here,
I've been iterating on Py 3.13 support and have it "almost" green.
https://github.com/apache/airflow/pull/46891
Currently (until connexion 2.15.0 release) - we cannot make it work with
FAB so FAB (Among others) is excluded.
However this brings the need to fix one more incompatibility (except that
we need to wait for at least RC release of the next provider's wave) -
serve_logs implicitly depends on Flask/Fab.
Currently our "serve logs" method to serve logs for celery workers is still
part of the "airflow-core" and indirectly depends on Flask/FAB APP being
installed.
There was a discussion that it should be possible to easily rewrite it
using fast-api, and it's the time we actually need it to:
* have Python 3.13 support
* become truly FAB-independent
I created an issue for it https://github.com/apache/airflow/issues/52526 -
and since I have not been involved in fast-api development, I guess it will
be faster by someone who was.
Can someone help with it ?
J.
On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> I also started to do more preparation and extracting things that we will
> need to merge first and release providers with some limits lifted. We need
> to make our providers released in PyPI prepared upfront because currently
> we are not able to generate PyPI constraints for Python 3.13 - I already
> have one of those merged - https://github.com/apache/airflow/pull/51994
> where we have to exclude `ray` from Python 3.13 and release provider before
> we get the PR "green" - similarly I have just created
> https://github.com/apache/airflow/pull/52060 to all providers that are
> currently limiting pandas (unnecessary it seems from my tests in 3.13 PR).
>
> J.
>
>
> On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <am...@gmail.com>
> wrote:
>
>> Thanks for taking this on Jarek.
>>
>> These are not easy tasks to work on and I cannot think of a better person
>> to do it than you!
>>
>> I will review it shortly this week.
>>
>> I think it is a good thing to introduce in Airflow 3.1 and I would not be
>> too much in favour
>> to cherry pick it to the 3.x series, just for convenience sake.
>>
>> Thanks & Regards,
>> Amogh Desai
>>
>>
>> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> > Hey here,
>> >
>> > Over the last few days and weeks I made significant progress with Python
>> > 3.13 support and while the PR is not yet fully green - we are getting
>> > close. I undrafted my PR recently:
>> >
>> > https://github.com/apache/airflow/pull/46891
>> >
>> > I have a kind request for others to start reviewing it, I will be happy
>> to
>> > respond to questions and I am also happy to accept some fixups to solve
>> > things in a better way if possible, I did a few hacks and left some
>> TODOs
>> > that might need some creative ideas to be solved in a better, simpler
>> way.
>> >
>> > I split the PR into several commits - and keep it continuously rebased
>> on
>> > top of the latest main. I had to remove some providers (including FAB -
>> > which is actually cool, because we can finally see that airflow
>> codebase is
>> > finally working even without FAB being around) - some of them are yet to
>> > implement Python 3.13 (for example Apache Beam is working on it) so we
>> > might add them back soon.
>> >
>> > Note - Python 3.13 was a WAY more difficult migration than Python 3.12 -
>> > there were a number of things removed/changed and small behavioral
>> changes
>> > that we relied on, also the dependencies are catching up way later, and
>> > there are some tricky dependencies of ours that make things more complex
>> > when it comes to selecting "min" versions of those. But we are finally
>> > getting there.
>> >
>> > Looks like we will be on-time to have 3.13 support for Airflow 3.1. We
>> > might also attempt to cherry-pick it for 3.0, but it might be too much
>> of a
>> > hassle, so we can decide after we merge this one to main.
>> >
>> > J.
>> >
>>
>
Re: Python 3.13 is coming
Posted by Jarek Potiuk <ja...@potiuk.com>.
And merged :).
Update, sync and you can start using all the goodies of Python 3.13 -
including much better Python REPL (colors!, sane editing) coloured
tracebacks, and the brave ones can try airflow with "free threading" (AKA
no GIL) - requires special python build and JIT (does not yet bring
performance improvements).
On Thu, Jul 17, 2025 at 2:37 PM Pavankumar Gopidesu <go...@gmail.com>
wrote:
> Verified the docs related changes, looking good :)
>
> Thanks Jarek, great work.
>
> Pavan
>
> On Thu, Jul 17, 2025 at 1:26 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > All right .. I addressed the next round of feedback and some
> investigations
> > where things were questioned led to interesting findings :)
> >
> > * interesting finding about having to flush() before you query.delete ->
> > that was accidentally revealed by removing fab plugins initialization in
> > Python 3.13 ->
> > https://github.com/apache/airflow/pull/46891#discussion_r2209325342
> > * Likely root cause (long standing Python bug with mmap that was fixed in
> > Python 3.13) for "too many open files" occasional issue people reported
> > recently with our supervisor/task creation ->
> > https://github.com/apache/airflow/pull/46891#discussion_r2209326571
> >
> > Both are pretty cool findings and the second might even lead to some
> > workarounds to improve number of opened file descriptors causing for
> > example this one: https://github.com/apache/airflow/issues/51512
> >
> > Thanks Vincent, Pavan, Amogh, Aritra, Jens for the reviews and some
> > "Hawk-eye" findings :) ... Really cool
> >
> > The PR is now heading to be green back after some experiments with fixing
> > things 'better" - and I plan to merge it before the dev call today - the
> > "weaviate" implementation can be likely more thoroughly tested with the
> > next provider's wave when we have an RC for weaviate provider - that will
> > be easier to test I guess (and it anyhow impacts really only Python 3.13
> > installation of Weaviate provider).
> >
> > If anyone would like to make another pass - let me know and I will
> hold-on
> > with merging.
> >
> > J.
> >
> >
> > On Thu, Jul 17, 2025 at 12:03 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > Cool. I have been spending more time with my family visiting me, so did
> > > not look yet at the comments - but I will - shortly - and will be happy
> > to
> > > merge it soon :D
> > >
> > > On Wed, Jul 16, 2025 at 9:51 AM Aritra Basu <ar...@gmail.com>
> > > wrote:
> > >
> > >> Great job on the PR Jarek! Looked through the broad strokes of the PR
> > and
> > >> all looked good to me.
> > >> --
> > >> Regards,
> > >> Aritra Basu
> > >>
> > >> On Wed, 16 Jul 2025, 11:38 am Amogh Desai, <am...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Jarek,
> > >> >
> > >> > That was quite a long PR to review. I focused most of my review time
> > on
> > >> the
> > >> > areas
> > >> > I’m more familiar with — (1) and (4). But I ended up going through
> the
> > >> rest
> > >> > as well 🙂
> > >> >
> > >> > Great job getting that PR green! I really appreciate the inline
> > comments
> > >> > with active TODOS,
> > >> > they are helpful and make it easier to follow the context.
> > >> >
> > >> > Thanks & Regards,
> > >> > Amogh Desai
> > >> >
> > >> >
> > >> > On Wed, Jul 16, 2025 at 1:47 AM Jens Scheffler
> > >> <j_scheffler@gmx.de.invalid
> > >> > >
> > >> > wrote:
> > >> >
> > >> > > Hi Jarek,
> > >> > >
> > >> > > cool! Thanks for all the efforts and the breath in preparing this!
> > >> Added
> > >> > > some comments in PR gut greatly appreciate (1) that notes describe
> > the
> > >> > > upper bindings and (2) that the reference image default stays with
> > >> 3.12
> > >> > > as 3.13 lags critical providers being ready.
> > >> > >
> > >> > > Jens
> > >> > >
> > >> > > On 15.07.25 19:52, Jarek Potiuk wrote:
> > >> > > > *TL;DR; Python 3.13 is ... ALMOST here... All you need is final
> > >> reviews
> > >> > > and
> > >> > > > merge !!!! *
> > >> > > >
> > >> > > > The PR is finally about to get green - after about 100 or so
> > >> iterations
> > >> > > (or
> > >> > > > so it seems - I opened it on Feb 19th (!) - so it's 5 months in
> > the
> > >> > > > making). https://github.com/apache/airflow/pull/46891. I would
> > >> really
> > >> > > > appreciate some final reviews and approvals.
> > >> > > >
> > >> > > > Jens and Pavan had already made a number of comments over the
> > >> loooong
> > >> > > > course of this one and I **think** I addressed all of them. I am
> > >> happy
> > >> > to
> > >> > > > get more comments - maybe some of the choices I made are
> outdated
> > or
> > >> > can
> > >> > > be
> > >> > > > improved - but now having the PR green it will be way easier to
> > >> iterate
> > >> > > on
> > >> > > > it
> > >> > > >
> > >> > > > It took a really long time and a number of iterations and some
> > >> > > opinionated
> > >> > > > choices. Mostly because FAB cannot be installed yet with Python
> > >> 3.13 -
> > >> > > but
> > >> > > > it also gave me an opportunity to move out or run conditionally
> > >> tests -
> > >> > > for
> > >> > > > all cases where FAB was still not completely removed from the
> > core.
> > >> One
> > >> > > of
> > >> > > > our tests (lowest-deps core Python 3.13) runs the complete suite
> > of
> > >> > tests
> > >> > > > for core where `fab-provider` and `flask-app-builder` are
> > completely
> > >> > > > missing from the CI environment (BTW. `uv sync` is an awesome
> > >> feature
> > >> > of
> > >> > > > `uv` and it made it possible). And a number of changes from
> other
> > >> > people
> > >> > > > (Jens, Pavan, LIU ZHE YOU, Wei Lee, Vincent, Elad and many
> others
> > >> had
> > >> > to
> > >> > > > be merged before we could get this one green).
> > >> > > >
> > >> > > > So I have a kind request - for those who know the codebase well
> -
> > >> > please
> > >> > > > take a look and let me know if you have any comments. Yes, it's
> a
> > >> > > monstrous
> > >> > > > PR and - unfortunately - it's not easily breakable into smaller
> > >> ones,
> > >> > > but I
> > >> > > > tried to make it as easy to review as possible. I split it into
> 4
> > >> > > > independent commits (which will still have to be squashed and
> > merged
> > >> > > > together):
> > >> > > >
> > >> > > > 1) Breeze changes for Python 3.13 - changes in breeze code that
> > >> enable
> > >> > > > Python 3.13
> > >> > > > 2) Pyproject.toml changes in Providers for Python 3.13
> (including
> > >> > > > dependencies and python_requires) - big number of files
> > >> > > > 3) README.rst changes for Providers for Python 3.13 (big number
> of
> > >> > files)
> > >> > > > 4) Actual changes that were necessary to get the PR green for
> > Python
> > >> > 3.13
> > >> > > > (smaller number, but more substantial changes in airflow-core
> and
> > >> > > provider
> > >> > > > code that makes Python 3.13 work)
> > >> > > >
> > >> > > > Currently excluded providers for Python 3.13:
> > >> > > > ** apache.beam*
> > >> > > > ** apache.kafka*
> > >> > > > ** fab*
> > >> > > > ** **yandex*
> > >> > > > ** ydb*
> > >> > > >
> > >> > > > They are excluded for various reasons - TODO:s left explaining
> the
> > >> > > reasons
> > >> > > > and we can gradually bring them in as blockers are removed).
> > >> Detailed
> > >> > > > summary of changes in the PR and copied below.
> > >> > > >
> > >> > > > When you review the code, you can focus on the part that you are
> > >> > > interested
> > >> > > > in (1) - if you want to look at breeze and (4) if you want to
> look
> > >> at
> > >> > > code
> > >> > > > changes, and skip rather easily absolutely necessary dependency
> > >> > > requirement
> > >> > > > changes in (2) and mostly-automated changes from (3).
> > >> > > >
> > >> > > > Yep I am asking for a lot, sorry, but I can't see how I could
> > break
> > >> > that
> > >> > > > one down into separate PRs easily.
> > >> > > >
> > >> > > > I have two kind requests to take a closer look specifically:
> > >> > > >
> > >> > > > a) for *Pavan* -> if you could check the aws-wrangler removal
> > >> replacing
> > >> > > it
> > >> > > > with botocore - for docs publishing. I am not 100% sure if I got
> > it
> > >> > > right,
> > >> > > > I did test it but a second pair of hands would be great to test
> it
> > >> > > >
> > >> > > > b) for *Wei Lee, Kevin Yang, and Rahul* *Vats* (since you were
> the
> > >> ones
> > >> > > > working on Waeviate in the past) -> I would really appreciate it
> > if
> > >> you
> > >> > > > could test the Weaviate changes I made. Weaviate was one of the
> > >> > > problematic
> > >> > > > changes because dependency update for 3.13 "freed" upgrade to a
> > >> newer
> > >> > > > version of Waeviate client (>=4.10.0) that used a httpx rather
> > than
> > >> > > > requests - and we had some low-level mock tests for retry
> > scenarios
> > >> > (429,
> > >> > > > 500, ConnectErrors). And in order to fix it, I had to not only
> > have
> > >> > code
> > >> > > in
> > >> > > > tests for mocking either httpx or requests, but I also had to
> > change
> > >> > the
> > >> > > > retriable error detection in Weaviate provider. I **think** I
> made
> > >> > proper
> > >> > > > changes there, but I have no experience/no Weaviate instance so
> I
> > >> had
> > >> > not
> > >> > > > tested it with the "real" weaviate - I updated the tests and
> > mocked
> > >> > httpx
> > >> > > > in the way I **think** it will work, but I am not 100% sure if I
> > >> got it
> > >> > > > right. Actual tests with weaviate would be great (and you had
> done
> > >> such
> > >> > > > tests before so it might be way easier for you).
> > >> > > >
> > >> > > > J.
> > >> > > >
> > >> > > > -------------
> > >> > > >
> > >> > > > This is the summary of the changes made. Beware .... it's .....
> > >> > > > LOOOONG.....
> > >> > > >
> > >> > > > Add Python 3.13 support for Airflow.
> > >> > > >
> > >> > > > Added Python 3.13 support across the codebase, including:
> > >> > > > * CI/CD workflow updates to handle Python 3.13.
> > >> > > > * Dockerfile and environment variable adjustments for
> > >> > compatibility.
> > >> > > > * Documentation updates to list Python 3.13 as supported.
> > >> > > > * Dependency and code changes for compatibility with
> > >> new/changed
> > >> > > > libraries (e.g., numpy 2.x, greenlet, pendulum).
> > >> > > > * Test and plugin manager adjustments for Python
> > 3.13-specific
> > >> > > > behaviors.
> > >> > > > * Removal or skipping of tests and features not compatible
> > with
> > >> > > Python
> > >> > > > 3.13.
> > >> > > > * Improved error handling and logging for unsupported
> > >> > > plugins/providers
> > >> > > > on Python 3.13.
> > >> > > > * Updated all provider README.rst files to:
> > >> > > > * List Python 3.13 as a supported version.
> > >> > > > * Adjust dependency requirements for Python 3.13 (e.g.,
> > >> pandas,
> > >> > > > pyarrow, ray, etc.).
> > >> > > > * Add or update notes about excluded or conditionally
> > >> supported
> > >> > > > dependencies for Python 3.13.
> > >> > > > * Document any known incompatibilities or workarounds for
> > >> > specific
> > >> > > > providers.
> > >> > > > * Updated all provider pyproject.toml files and automation
> > to:
> > >> > > > * Add Python 3.13 to the supported classifiers.
> > >> > > > * Adjust requires-python and dependency constraints for
> > >> Python
> > >> > > 3.13.
> > >> > > > * Exclude Python 3.13 for providers that are not yet
> > >> compatible
> > >> > > (e.g.,
> > >> > > > those depending on Flask AppBuilder).
> > >> > > > * Add conditional dependencies for new versions of
> > libraries
> > >> > > required
> > >> > > > by Python 3.13.
> > >> > > > * Updated Breeze and related scripts to:
> > >> > > > * Build and test with Python 3.13.
> > >> > > > * Update documentation and release scripts to include
> Python
> > >> > 3.13.
> > >> > > > * Ensure all dev tooling and test environments work with
> > >> Python
> > >> > > 3.13.
> > >> > > > * Removed remaining FAB related code from core and removed
> > >> tests
> > >> > for
> > >> > > > older FAB provider versions (<1.3.0)
> > >> > > > * some of the tests in core still depend on FAB - but we
> make
> > >> them
> > >> > > > optional - both for parsing (we need to skip some
> imports)
> > >> and
> > >> > > > execution - because they test a legacy behaviour of
> > embedding
> > >> > > > Flask plugins that is still supported in core.
> > >> > > > * k8s tests now run also with SimpleAuthManager and
> > >> > > SimpleAuthManager
> > >> > > > is used automatically when running tests in Python 3.13
> > >> > > > * migration tests had to be excluded for Python 3.13
> because
> > we
> > >> > > cannot
> > >> > > > migrate down on Python 3.13 without FAB tables created.
> The
> > >> > tests
> > >> > > > will be brought back when FAB is supported or when we
> > release
> > >> > 3.1
> > >> > > > and start working on 3.2, we should be able to migrate
> down
> > >> to
> > >> > 3.1
> > >> > > > * Weaviate tests have been updated to handle case where
> > >> > > weaviate-client
> > >> > > > is > 2.10.0 - because it started to use httpx instead of
> > >> > requests,
> > >> > > > and Python 3.13 actually unblocks using older version of
> > >> > weaviate
> > >> > > > (it was previously blocked indirectly via Flask
> > dependencies
> > >> and
> > >> > > > grpcio).
> > >> > > > * Added doc change in our rules on what is the "default"
> > Python
> > >> > > version
> > >> > > > for airflow images - handling the case where we cannot
> use
> > >> > > "newest"
> > >> > > > Python version as default where not all "regular" image
> > >> > providers
> > >> > > are
> > >> > > > supported in the latest version
> > >> > > >
> > >> > > > J.
> > >> > > >
> > >> > > > On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com>
> > >> wrote:
> > >> > > >
> > >> > > >> I also started to do more preparation and extracting things
> that
> > we
> > >> > will
> > >> > > >> need to merge first and release providers with some limits
> > lifted.
> > >> We
> > >> > > need
> > >> > > >> to make our providers released in PyPI prepared upfront because
> > >> > > currently
> > >> > > >> we are not able to generate PyPI constraints for Python 3.13 -
> I
> > >> > already
> > >> > > >> have one of those merged -
> > >> > https://github.com/apache/airflow/pull/51994
> > >> > > >> where we have to exclude `ray` from Python 3.13 and release
> > >> provider
> > >> > > before
> > >> > > >> we get the PR "green" - similarly I have just created
> > >> > > >> https://github.com/apache/airflow/pull/52060 to all providers
> > that
> > >> > are
> > >> > > >> currently limiting pandas (unnecessary it seems from my tests
> in
> > >> 3.13
> > >> > > PR).
> > >> > > >>
> > >> > > >> J.
> > >> > > >>
> > >> > > >>
> > >> > > >> On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <
> > >> amoghdesai.oss@gmail.com
> > >> > >
> > >> > > >> wrote:
> > >> > > >>
> > >> > > >>> Thanks for taking this on Jarek.
> > >> > > >>>
> > >> > > >>> These are not easy tasks to work on and I cannot think of a
> > better
> > >> > > person
> > >> > > >>> to do it than you!
> > >> > > >>>
> > >> > > >>> I will review it shortly this week.
> > >> > > >>>
> > >> > > >>> I think it is a good thing to introduce in Airflow 3.1 and I
> > would
> > >> > not
> > >> > > be
> > >> > > >>> too much in favour
> > >> > > >>> to cherry pick it to the 3.x series, just for convenience
> sake.
> > >> > > >>>
> > >> > > >>> Thanks & Regards,
> > >> > > >>> Amogh Desai
> > >> > > >>>
> > >> > > >>>
> > >> > > >>> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <
> jarek@potiuk.com>
> > >> > wrote:
> > >> > > >>>
> > >> > > >>>> Hey here,
> > >> > > >>>>
> > >> > > >>>> Over the last few days and weeks I made significant progress
> > with
> > >> > > Python
> > >> > > >>>> 3.13 support and while the PR is not yet fully green - we are
> > >> > getting
> > >> > > >>>> close. I undrafted my PR recently:
> > >> > > >>>>
> > >> > > >>>> https://github.com/apache/airflow/pull/46891
> > >> > > >>>>
> > >> > > >>>> I have a kind request for others to start reviewing it, I
> will
> > be
> > >> > > happy
> > >> > > >>> to
> > >> > > >>>> respond to questions and I am also happy to accept some
> fixups
> > to
> > >> > > solve
> > >> > > >>>> things in a better way if possible, I did a few hacks and
> left
> > >> some
> > >> > > >>> TODOs
> > >> > > >>>> that might need some creative ideas to be solved in a better,
> > >> > simpler
> > >> > > >>> way.
> > >> > > >>>> I split the PR into several commits - and keep it
> continuously
> > >> > rebased
> > >> > > >>> on
> > >> > > >>>> top of the latest main. I had to remove some providers
> > (including
> > >> > FAB
> > >> > > -
> > >> > > >>>> which is actually cool, because we can finally see that
> airflow
> > >> > > >>> codebase is
> > >> > > >>>> finally working even without FAB being around) - some of them
> > are
> > >> > yet
> > >> > > to
> > >> > > >>>> implement Python 3.13 (for example Apache Beam is working on
> > it)
> > >> so
> > >> > we
> > >> > > >>>> might add them back soon.
> > >> > > >>>>
> > >> > > >>>> Note - Python 3.13 was a WAY more difficult migration than
> > Python
> > >> > > 3.12 -
> > >> > > >>>> there were a number of things removed/changed and small
> > >> behavioral
> > >> > > >>> changes
> > >> > > >>>> that we relied on, also the dependencies are catching up way
> > >> later,
> > >> > > and
> > >> > > >>>> there are some tricky dependencies of ours that make things
> > more
> > >> > > complex
> > >> > > >>>> when it comes to selecting "min" versions of those. But we
> are
> > >> > finally
> > >> > > >>>> getting there.
> > >> > > >>>>
> > >> > > >>>> Looks like we will be on-time to have 3.13 support for
> Airflow
> > >> 3.1.
> > >> > We
> > >> > > >>>> might also attempt to cherry-pick it for 3.0, but it might be
> > too
> > >> > much
> > >> > > >>> of a
> > >> > > >>>> hassle, so we can decide after we merge this one to main.
> > >> > > >>>>
> > >> > > >>>> J.
> > >> > > >>>>
> > >> > >
> > >> > >
> > ---------------------------------------------------------------------
> > >> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: Python 3.13 is coming
Posted by Pavankumar Gopidesu <go...@gmail.com>.
Verified the docs related changes, looking good :)
Thanks Jarek, great work.
Pavan
On Thu, Jul 17, 2025 at 1:26 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> All right .. I addressed the next round of feedback and some investigations
> where things were questioned led to interesting findings :)
>
> * interesting finding about having to flush() before you query.delete ->
> that was accidentally revealed by removing fab plugins initialization in
> Python 3.13 ->
> https://github.com/apache/airflow/pull/46891#discussion_r2209325342
> * Likely root cause (long standing Python bug with mmap that was fixed in
> Python 3.13) for "too many open files" occasional issue people reported
> recently with our supervisor/task creation ->
> https://github.com/apache/airflow/pull/46891#discussion_r2209326571
>
> Both are pretty cool findings and the second might even lead to some
> workarounds to improve number of opened file descriptors causing for
> example this one: https://github.com/apache/airflow/issues/51512
>
> Thanks Vincent, Pavan, Amogh, Aritra, Jens for the reviews and some
> "Hawk-eye" findings :) ... Really cool
>
> The PR is now heading to be green back after some experiments with fixing
> things 'better" - and I plan to merge it before the dev call today - the
> "weaviate" implementation can be likely more thoroughly tested with the
> next provider's wave when we have an RC for weaviate provider - that will
> be easier to test I guess (and it anyhow impacts really only Python 3.13
> installation of Weaviate provider).
>
> If anyone would like to make another pass - let me know and I will hold-on
> with merging.
>
> J.
>
>
> On Thu, Jul 17, 2025 at 12:03 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Cool. I have been spending more time with my family visiting me, so did
> > not look yet at the comments - but I will - shortly - and will be happy
> to
> > merge it soon :D
> >
> > On Wed, Jul 16, 2025 at 9:51 AM Aritra Basu <ar...@gmail.com>
> > wrote:
> >
> >> Great job on the PR Jarek! Looked through the broad strokes of the PR
> and
> >> all looked good to me.
> >> --
> >> Regards,
> >> Aritra Basu
> >>
> >> On Wed, 16 Jul 2025, 11:38 am Amogh Desai, <am...@gmail.com>
> >> wrote:
> >>
> >> > Hi Jarek,
> >> >
> >> > That was quite a long PR to review. I focused most of my review time
> on
> >> the
> >> > areas
> >> > I’m more familiar with — (1) and (4). But I ended up going through the
> >> rest
> >> > as well 🙂
> >> >
> >> > Great job getting that PR green! I really appreciate the inline
> comments
> >> > with active TODOS,
> >> > they are helpful and make it easier to follow the context.
> >> >
> >> > Thanks & Regards,
> >> > Amogh Desai
> >> >
> >> >
> >> > On Wed, Jul 16, 2025 at 1:47 AM Jens Scheffler
> >> <j_scheffler@gmx.de.invalid
> >> > >
> >> > wrote:
> >> >
> >> > > Hi Jarek,
> >> > >
> >> > > cool! Thanks for all the efforts and the breath in preparing this!
> >> Added
> >> > > some comments in PR gut greatly appreciate (1) that notes describe
> the
> >> > > upper bindings and (2) that the reference image default stays with
> >> 3.12
> >> > > as 3.13 lags critical providers being ready.
> >> > >
> >> > > Jens
> >> > >
> >> > > On 15.07.25 19:52, Jarek Potiuk wrote:
> >> > > > *TL;DR; Python 3.13 is ... ALMOST here... All you need is final
> >> reviews
> >> > > and
> >> > > > merge !!!! *
> >> > > >
> >> > > > The PR is finally about to get green - after about 100 or so
> >> iterations
> >> > > (or
> >> > > > so it seems - I opened it on Feb 19th (!) - so it's 5 months in
> the
> >> > > > making). https://github.com/apache/airflow/pull/46891. I would
> >> really
> >> > > > appreciate some final reviews and approvals.
> >> > > >
> >> > > > Jens and Pavan had already made a number of comments over the
> >> loooong
> >> > > > course of this one and I **think** I addressed all of them. I am
> >> happy
> >> > to
> >> > > > get more comments - maybe some of the choices I made are outdated
> or
> >> > can
> >> > > be
> >> > > > improved - but now having the PR green it will be way easier to
> >> iterate
> >> > > on
> >> > > > it
> >> > > >
> >> > > > It took a really long time and a number of iterations and some
> >> > > opinionated
> >> > > > choices. Mostly because FAB cannot be installed yet with Python
> >> 3.13 -
> >> > > but
> >> > > > it also gave me an opportunity to move out or run conditionally
> >> tests -
> >> > > for
> >> > > > all cases where FAB was still not completely removed from the
> core.
> >> One
> >> > > of
> >> > > > our tests (lowest-deps core Python 3.13) runs the complete suite
> of
> >> > tests
> >> > > > for core where `fab-provider` and `flask-app-builder` are
> completely
> >> > > > missing from the CI environment (BTW. `uv sync` is an awesome
> >> feature
> >> > of
> >> > > > `uv` and it made it possible). And a number of changes from other
> >> > people
> >> > > > (Jens, Pavan, LIU ZHE YOU, Wei Lee, Vincent, Elad and many others
> >> had
> >> > to
> >> > > > be merged before we could get this one green).
> >> > > >
> >> > > > So I have a kind request - for those who know the codebase well -
> >> > please
> >> > > > take a look and let me know if you have any comments. Yes, it's a
> >> > > monstrous
> >> > > > PR and - unfortunately - it's not easily breakable into smaller
> >> ones,
> >> > > but I
> >> > > > tried to make it as easy to review as possible. I split it into 4
> >> > > > independent commits (which will still have to be squashed and
> merged
> >> > > > together):
> >> > > >
> >> > > > 1) Breeze changes for Python 3.13 - changes in breeze code that
> >> enable
> >> > > > Python 3.13
> >> > > > 2) Pyproject.toml changes in Providers for Python 3.13 (including
> >> > > > dependencies and python_requires) - big number of files
> >> > > > 3) README.rst changes for Providers for Python 3.13 (big number of
> >> > files)
> >> > > > 4) Actual changes that were necessary to get the PR green for
> Python
> >> > 3.13
> >> > > > (smaller number, but more substantial changes in airflow-core and
> >> > > provider
> >> > > > code that makes Python 3.13 work)
> >> > > >
> >> > > > Currently excluded providers for Python 3.13:
> >> > > > ** apache.beam*
> >> > > > ** apache.kafka*
> >> > > > ** fab*
> >> > > > ** **yandex*
> >> > > > ** ydb*
> >> > > >
> >> > > > They are excluded for various reasons - TODO:s left explaining the
> >> > > reasons
> >> > > > and we can gradually bring them in as blockers are removed).
> >> Detailed
> >> > > > summary of changes in the PR and copied below.
> >> > > >
> >> > > > When you review the code, you can focus on the part that you are
> >> > > interested
> >> > > > in (1) - if you want to look at breeze and (4) if you want to look
> >> at
> >> > > code
> >> > > > changes, and skip rather easily absolutely necessary dependency
> >> > > requirement
> >> > > > changes in (2) and mostly-automated changes from (3).
> >> > > >
> >> > > > Yep I am asking for a lot, sorry, but I can't see how I could
> break
> >> > that
> >> > > > one down into separate PRs easily.
> >> > > >
> >> > > > I have two kind requests to take a closer look specifically:
> >> > > >
> >> > > > a) for *Pavan* -> if you could check the aws-wrangler removal
> >> replacing
> >> > > it
> >> > > > with botocore - for docs publishing. I am not 100% sure if I got
> it
> >> > > right,
> >> > > > I did test it but a second pair of hands would be great to test it
> >> > > >
> >> > > > b) for *Wei Lee, Kevin Yang, and Rahul* *Vats* (since you were the
> >> ones
> >> > > > working on Waeviate in the past) -> I would really appreciate it
> if
> >> you
> >> > > > could test the Weaviate changes I made. Weaviate was one of the
> >> > > problematic
> >> > > > changes because dependency update for 3.13 "freed" upgrade to a
> >> newer
> >> > > > version of Waeviate client (>=4.10.0) that used a httpx rather
> than
> >> > > > requests - and we had some low-level mock tests for retry
> scenarios
> >> > (429,
> >> > > > 500, ConnectErrors). And in order to fix it, I had to not only
> have
> >> > code
> >> > > in
> >> > > > tests for mocking either httpx or requests, but I also had to
> change
> >> > the
> >> > > > retriable error detection in Weaviate provider. I **think** I made
> >> > proper
> >> > > > changes there, but I have no experience/no Weaviate instance so I
> >> had
> >> > not
> >> > > > tested it with the "real" weaviate - I updated the tests and
> mocked
> >> > httpx
> >> > > > in the way I **think** it will work, but I am not 100% sure if I
> >> got it
> >> > > > right. Actual tests with weaviate would be great (and you had done
> >> such
> >> > > > tests before so it might be way easier for you).
> >> > > >
> >> > > > J.
> >> > > >
> >> > > > -------------
> >> > > >
> >> > > > This is the summary of the changes made. Beware .... it's .....
> >> > > > LOOOONG.....
> >> > > >
> >> > > > Add Python 3.13 support for Airflow.
> >> > > >
> >> > > > Added Python 3.13 support across the codebase, including:
> >> > > > * CI/CD workflow updates to handle Python 3.13.
> >> > > > * Dockerfile and environment variable adjustments for
> >> > compatibility.
> >> > > > * Documentation updates to list Python 3.13 as supported.
> >> > > > * Dependency and code changes for compatibility with
> >> new/changed
> >> > > > libraries (e.g., numpy 2.x, greenlet, pendulum).
> >> > > > * Test and plugin manager adjustments for Python
> 3.13-specific
> >> > > > behaviors.
> >> > > > * Removal or skipping of tests and features not compatible
> with
> >> > > Python
> >> > > > 3.13.
> >> > > > * Improved error handling and logging for unsupported
> >> > > plugins/providers
> >> > > > on Python 3.13.
> >> > > > * Updated all provider README.rst files to:
> >> > > > * List Python 3.13 as a supported version.
> >> > > > * Adjust dependency requirements for Python 3.13 (e.g.,
> >> pandas,
> >> > > > pyarrow, ray, etc.).
> >> > > > * Add or update notes about excluded or conditionally
> >> supported
> >> > > > dependencies for Python 3.13.
> >> > > > * Document any known incompatibilities or workarounds for
> >> > specific
> >> > > > providers.
> >> > > > * Updated all provider pyproject.toml files and automation
> to:
> >> > > > * Add Python 3.13 to the supported classifiers.
> >> > > > * Adjust requires-python and dependency constraints for
> >> Python
> >> > > 3.13.
> >> > > > * Exclude Python 3.13 for providers that are not yet
> >> compatible
> >> > > (e.g.,
> >> > > > those depending on Flask AppBuilder).
> >> > > > * Add conditional dependencies for new versions of
> libraries
> >> > > required
> >> > > > by Python 3.13.
> >> > > > * Updated Breeze and related scripts to:
> >> > > > * Build and test with Python 3.13.
> >> > > > * Update documentation and release scripts to include Python
> >> > 3.13.
> >> > > > * Ensure all dev tooling and test environments work with
> >> Python
> >> > > 3.13.
> >> > > > * Removed remaining FAB related code from core and removed
> >> tests
> >> > for
> >> > > > older FAB provider versions (<1.3.0)
> >> > > > * some of the tests in core still depend on FAB - but we make
> >> them
> >> > > > optional - both for parsing (we need to skip some imports)
> >> and
> >> > > > execution - because they test a legacy behaviour of
> embedding
> >> > > > Flask plugins that is still supported in core.
> >> > > > * k8s tests now run also with SimpleAuthManager and
> >> > > SimpleAuthManager
> >> > > > is used automatically when running tests in Python 3.13
> >> > > > * migration tests had to be excluded for Python 3.13 because
> we
> >> > > cannot
> >> > > > migrate down on Python 3.13 without FAB tables created. The
> >> > tests
> >> > > > will be brought back when FAB is supported or when we
> release
> >> > 3.1
> >> > > > and start working on 3.2, we should be able to migrate down
> >> to
> >> > 3.1
> >> > > > * Weaviate tests have been updated to handle case where
> >> > > weaviate-client
> >> > > > is > 2.10.0 - because it started to use httpx instead of
> >> > requests,
> >> > > > and Python 3.13 actually unblocks using older version of
> >> > weaviate
> >> > > > (it was previously blocked indirectly via Flask
> dependencies
> >> and
> >> > > > grpcio).
> >> > > > * Added doc change in our rules on what is the "default"
> Python
> >> > > version
> >> > > > for airflow images - handling the case where we cannot use
> >> > > "newest"
> >> > > > Python version as default where not all "regular" image
> >> > providers
> >> > > are
> >> > > > supported in the latest version
> >> > > >
> >> > > > J.
> >> > > >
> >> > > > On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com>
> >> wrote:
> >> > > >
> >> > > >> I also started to do more preparation and extracting things that
> we
> >> > will
> >> > > >> need to merge first and release providers with some limits
> lifted.
> >> We
> >> > > need
> >> > > >> to make our providers released in PyPI prepared upfront because
> >> > > currently
> >> > > >> we are not able to generate PyPI constraints for Python 3.13 - I
> >> > already
> >> > > >> have one of those merged -
> >> > https://github.com/apache/airflow/pull/51994
> >> > > >> where we have to exclude `ray` from Python 3.13 and release
> >> provider
> >> > > before
> >> > > >> we get the PR "green" - similarly I have just created
> >> > > >> https://github.com/apache/airflow/pull/52060 to all providers
> that
> >> > are
> >> > > >> currently limiting pandas (unnecessary it seems from my tests in
> >> 3.13
> >> > > PR).
> >> > > >>
> >> > > >> J.
> >> > > >>
> >> > > >>
> >> > > >> On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <
> >> amoghdesai.oss@gmail.com
> >> > >
> >> > > >> wrote:
> >> > > >>
> >> > > >>> Thanks for taking this on Jarek.
> >> > > >>>
> >> > > >>> These are not easy tasks to work on and I cannot think of a
> better
> >> > > person
> >> > > >>> to do it than you!
> >> > > >>>
> >> > > >>> I will review it shortly this week.
> >> > > >>>
> >> > > >>> I think it is a good thing to introduce in Airflow 3.1 and I
> would
> >> > not
> >> > > be
> >> > > >>> too much in favour
> >> > > >>> to cherry pick it to the 3.x series, just for convenience sake.
> >> > > >>>
> >> > > >>> Thanks & Regards,
> >> > > >>> Amogh Desai
> >> > > >>>
> >> > > >>>
> >> > > >>> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com>
> >> > wrote:
> >> > > >>>
> >> > > >>>> Hey here,
> >> > > >>>>
> >> > > >>>> Over the last few days and weeks I made significant progress
> with
> >> > > Python
> >> > > >>>> 3.13 support and while the PR is not yet fully green - we are
> >> > getting
> >> > > >>>> close. I undrafted my PR recently:
> >> > > >>>>
> >> > > >>>> https://github.com/apache/airflow/pull/46891
> >> > > >>>>
> >> > > >>>> I have a kind request for others to start reviewing it, I will
> be
> >> > > happy
> >> > > >>> to
> >> > > >>>> respond to questions and I am also happy to accept some fixups
> to
> >> > > solve
> >> > > >>>> things in a better way if possible, I did a few hacks and left
> >> some
> >> > > >>> TODOs
> >> > > >>>> that might need some creative ideas to be solved in a better,
> >> > simpler
> >> > > >>> way.
> >> > > >>>> I split the PR into several commits - and keep it continuously
> >> > rebased
> >> > > >>> on
> >> > > >>>> top of the latest main. I had to remove some providers
> (including
> >> > FAB
> >> > > -
> >> > > >>>> which is actually cool, because we can finally see that airflow
> >> > > >>> codebase is
> >> > > >>>> finally working even without FAB being around) - some of them
> are
> >> > yet
> >> > > to
> >> > > >>>> implement Python 3.13 (for example Apache Beam is working on
> it)
> >> so
> >> > we
> >> > > >>>> might add them back soon.
> >> > > >>>>
> >> > > >>>> Note - Python 3.13 was a WAY more difficult migration than
> Python
> >> > > 3.12 -
> >> > > >>>> there were a number of things removed/changed and small
> >> behavioral
> >> > > >>> changes
> >> > > >>>> that we relied on, also the dependencies are catching up way
> >> later,
> >> > > and
> >> > > >>>> there are some tricky dependencies of ours that make things
> more
> >> > > complex
> >> > > >>>> when it comes to selecting "min" versions of those. But we are
> >> > finally
> >> > > >>>> getting there.
> >> > > >>>>
> >> > > >>>> Looks like we will be on-time to have 3.13 support for Airflow
> >> 3.1.
> >> > We
> >> > > >>>> might also attempt to cherry-pick it for 3.0, but it might be
> too
> >> > much
> >> > > >>> of a
> >> > > >>>> hassle, so we can decide after we merge this one to main.
> >> > > >>>>
> >> > > >>>> J.
> >> > > >>>>
> >> > >
> >> > >
> ---------------------------------------------------------------------
> >> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> > > For additional commands, e-mail: dev-help@airflow.apache.org
> >> > >
> >> > >
> >> >
> >>
> >
>
Re: Python 3.13 is coming
Posted by Jarek Potiuk <ja...@potiuk.com>.
All right .. I addressed the next round of feedback and some investigations
where things were questioned led to interesting findings :)
* interesting finding about having to flush() before you query.delete ->
that was accidentally revealed by removing fab plugins initialization in
Python 3.13 ->
https://github.com/apache/airflow/pull/46891#discussion_r2209325342
* Likely root cause (long standing Python bug with mmap that was fixed in
Python 3.13) for "too many open files" occasional issue people reported
recently with our supervisor/task creation ->
https://github.com/apache/airflow/pull/46891#discussion_r2209326571
Both are pretty cool findings and the second might even lead to some
workarounds to improve number of opened file descriptors causing for
example this one: https://github.com/apache/airflow/issues/51512
Thanks Vincent, Pavan, Amogh, Aritra, Jens for the reviews and some
"Hawk-eye" findings :) ... Really cool
The PR is now heading to be green back after some experiments with fixing
things 'better" - and I plan to merge it before the dev call today - the
"weaviate" implementation can be likely more thoroughly tested with the
next provider's wave when we have an RC for weaviate provider - that will
be easier to test I guess (and it anyhow impacts really only Python 3.13
installation of Weaviate provider).
If anyone would like to make another pass - let me know and I will hold-on
with merging.
J.
On Thu, Jul 17, 2025 at 12:03 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> Cool. I have been spending more time with my family visiting me, so did
> not look yet at the comments - but I will - shortly - and will be happy to
> merge it soon :D
>
> On Wed, Jul 16, 2025 at 9:51 AM Aritra Basu <ar...@gmail.com>
> wrote:
>
>> Great job on the PR Jarek! Looked through the broad strokes of the PR and
>> all looked good to me.
>> --
>> Regards,
>> Aritra Basu
>>
>> On Wed, 16 Jul 2025, 11:38 am Amogh Desai, <am...@gmail.com>
>> wrote:
>>
>> > Hi Jarek,
>> >
>> > That was quite a long PR to review. I focused most of my review time on
>> the
>> > areas
>> > I’m more familiar with — (1) and (4). But I ended up going through the
>> rest
>> > as well 🙂
>> >
>> > Great job getting that PR green! I really appreciate the inline comments
>> > with active TODOS,
>> > they are helpful and make it easier to follow the context.
>> >
>> > Thanks & Regards,
>> > Amogh Desai
>> >
>> >
>> > On Wed, Jul 16, 2025 at 1:47 AM Jens Scheffler
>> <j_scheffler@gmx.de.invalid
>> > >
>> > wrote:
>> >
>> > > Hi Jarek,
>> > >
>> > > cool! Thanks for all the efforts and the breath in preparing this!
>> Added
>> > > some comments in PR gut greatly appreciate (1) that notes describe the
>> > > upper bindings and (2) that the reference image default stays with
>> 3.12
>> > > as 3.13 lags critical providers being ready.
>> > >
>> > > Jens
>> > >
>> > > On 15.07.25 19:52, Jarek Potiuk wrote:
>> > > > *TL;DR; Python 3.13 is ... ALMOST here... All you need is final
>> reviews
>> > > and
>> > > > merge !!!! *
>> > > >
>> > > > The PR is finally about to get green - after about 100 or so
>> iterations
>> > > (or
>> > > > so it seems - I opened it on Feb 19th (!) - so it's 5 months in the
>> > > > making). https://github.com/apache/airflow/pull/46891. I would
>> really
>> > > > appreciate some final reviews and approvals.
>> > > >
>> > > > Jens and Pavan had already made a number of comments over the
>> loooong
>> > > > course of this one and I **think** I addressed all of them. I am
>> happy
>> > to
>> > > > get more comments - maybe some of the choices I made are outdated or
>> > can
>> > > be
>> > > > improved - but now having the PR green it will be way easier to
>> iterate
>> > > on
>> > > > it
>> > > >
>> > > > It took a really long time and a number of iterations and some
>> > > opinionated
>> > > > choices. Mostly because FAB cannot be installed yet with Python
>> 3.13 -
>> > > but
>> > > > it also gave me an opportunity to move out or run conditionally
>> tests -
>> > > for
>> > > > all cases where FAB was still not completely removed from the core.
>> One
>> > > of
>> > > > our tests (lowest-deps core Python 3.13) runs the complete suite of
>> > tests
>> > > > for core where `fab-provider` and `flask-app-builder` are completely
>> > > > missing from the CI environment (BTW. `uv sync` is an awesome
>> feature
>> > of
>> > > > `uv` and it made it possible). And a number of changes from other
>> > people
>> > > > (Jens, Pavan, LIU ZHE YOU, Wei Lee, Vincent, Elad and many others
>> had
>> > to
>> > > > be merged before we could get this one green).
>> > > >
>> > > > So I have a kind request - for those who know the codebase well -
>> > please
>> > > > take a look and let me know if you have any comments. Yes, it's a
>> > > monstrous
>> > > > PR and - unfortunately - it's not easily breakable into smaller
>> ones,
>> > > but I
>> > > > tried to make it as easy to review as possible. I split it into 4
>> > > > independent commits (which will still have to be squashed and merged
>> > > > together):
>> > > >
>> > > > 1) Breeze changes for Python 3.13 - changes in breeze code that
>> enable
>> > > > Python 3.13
>> > > > 2) Pyproject.toml changes in Providers for Python 3.13 (including
>> > > > dependencies and python_requires) - big number of files
>> > > > 3) README.rst changes for Providers for Python 3.13 (big number of
>> > files)
>> > > > 4) Actual changes that were necessary to get the PR green for Python
>> > 3.13
>> > > > (smaller number, but more substantial changes in airflow-core and
>> > > provider
>> > > > code that makes Python 3.13 work)
>> > > >
>> > > > Currently excluded providers for Python 3.13:
>> > > > ** apache.beam*
>> > > > ** apache.kafka*
>> > > > ** fab*
>> > > > ** **yandex*
>> > > > ** ydb*
>> > > >
>> > > > They are excluded for various reasons - TODO:s left explaining the
>> > > reasons
>> > > > and we can gradually bring them in as blockers are removed).
>> Detailed
>> > > > summary of changes in the PR and copied below.
>> > > >
>> > > > When you review the code, you can focus on the part that you are
>> > > interested
>> > > > in (1) - if you want to look at breeze and (4) if you want to look
>> at
>> > > code
>> > > > changes, and skip rather easily absolutely necessary dependency
>> > > requirement
>> > > > changes in (2) and mostly-automated changes from (3).
>> > > >
>> > > > Yep I am asking for a lot, sorry, but I can't see how I could break
>> > that
>> > > > one down into separate PRs easily.
>> > > >
>> > > > I have two kind requests to take a closer look specifically:
>> > > >
>> > > > a) for *Pavan* -> if you could check the aws-wrangler removal
>> replacing
>> > > it
>> > > > with botocore - for docs publishing. I am not 100% sure if I got it
>> > > right,
>> > > > I did test it but a second pair of hands would be great to test it
>> > > >
>> > > > b) for *Wei Lee, Kevin Yang, and Rahul* *Vats* (since you were the
>> ones
>> > > > working on Waeviate in the past) -> I would really appreciate it if
>> you
>> > > > could test the Weaviate changes I made. Weaviate was one of the
>> > > problematic
>> > > > changes because dependency update for 3.13 "freed" upgrade to a
>> newer
>> > > > version of Waeviate client (>=4.10.0) that used a httpx rather than
>> > > > requests - and we had some low-level mock tests for retry scenarios
>> > (429,
>> > > > 500, ConnectErrors). And in order to fix it, I had to not only have
>> > code
>> > > in
>> > > > tests for mocking either httpx or requests, but I also had to change
>> > the
>> > > > retriable error detection in Weaviate provider. I **think** I made
>> > proper
>> > > > changes there, but I have no experience/no Weaviate instance so I
>> had
>> > not
>> > > > tested it with the "real" weaviate - I updated the tests and mocked
>> > httpx
>> > > > in the way I **think** it will work, but I am not 100% sure if I
>> got it
>> > > > right. Actual tests with weaviate would be great (and you had done
>> such
>> > > > tests before so it might be way easier for you).
>> > > >
>> > > > J.
>> > > >
>> > > > -------------
>> > > >
>> > > > This is the summary of the changes made. Beware .... it's .....
>> > > > LOOOONG.....
>> > > >
>> > > > Add Python 3.13 support for Airflow.
>> > > >
>> > > > Added Python 3.13 support across the codebase, including:
>> > > > * CI/CD workflow updates to handle Python 3.13.
>> > > > * Dockerfile and environment variable adjustments for
>> > compatibility.
>> > > > * Documentation updates to list Python 3.13 as supported.
>> > > > * Dependency and code changes for compatibility with
>> new/changed
>> > > > libraries (e.g., numpy 2.x, greenlet, pendulum).
>> > > > * Test and plugin manager adjustments for Python 3.13-specific
>> > > > behaviors.
>> > > > * Removal or skipping of tests and features not compatible with
>> > > Python
>> > > > 3.13.
>> > > > * Improved error handling and logging for unsupported
>> > > plugins/providers
>> > > > on Python 3.13.
>> > > > * Updated all provider README.rst files to:
>> > > > * List Python 3.13 as a supported version.
>> > > > * Adjust dependency requirements for Python 3.13 (e.g.,
>> pandas,
>> > > > pyarrow, ray, etc.).
>> > > > * Add or update notes about excluded or conditionally
>> supported
>> > > > dependencies for Python 3.13.
>> > > > * Document any known incompatibilities or workarounds for
>> > specific
>> > > > providers.
>> > > > * Updated all provider pyproject.toml files and automation to:
>> > > > * Add Python 3.13 to the supported classifiers.
>> > > > * Adjust requires-python and dependency constraints for
>> Python
>> > > 3.13.
>> > > > * Exclude Python 3.13 for providers that are not yet
>> compatible
>> > > (e.g.,
>> > > > those depending on Flask AppBuilder).
>> > > > * Add conditional dependencies for new versions of libraries
>> > > required
>> > > > by Python 3.13.
>> > > > * Updated Breeze and related scripts to:
>> > > > * Build and test with Python 3.13.
>> > > > * Update documentation and release scripts to include Python
>> > 3.13.
>> > > > * Ensure all dev tooling and test environments work with
>> Python
>> > > 3.13.
>> > > > * Removed remaining FAB related code from core and removed
>> tests
>> > for
>> > > > older FAB provider versions (<1.3.0)
>> > > > * some of the tests in core still depend on FAB - but we make
>> them
>> > > > optional - both for parsing (we need to skip some imports)
>> and
>> > > > execution - because they test a legacy behaviour of embedding
>> > > > Flask plugins that is still supported in core.
>> > > > * k8s tests now run also with SimpleAuthManager and
>> > > SimpleAuthManager
>> > > > is used automatically when running tests in Python 3.13
>> > > > * migration tests had to be excluded for Python 3.13 because we
>> > > cannot
>> > > > migrate down on Python 3.13 without FAB tables created. The
>> > tests
>> > > > will be brought back when FAB is supported or when we release
>> > 3.1
>> > > > and start working on 3.2, we should be able to migrate down
>> to
>> > 3.1
>> > > > * Weaviate tests have been updated to handle case where
>> > > weaviate-client
>> > > > is > 2.10.0 - because it started to use httpx instead of
>> > requests,
>> > > > and Python 3.13 actually unblocks using older version of
>> > weaviate
>> > > > (it was previously blocked indirectly via Flask dependencies
>> and
>> > > > grpcio).
>> > > > * Added doc change in our rules on what is the "default" Python
>> > > version
>> > > > for airflow images - handling the case where we cannot use
>> > > "newest"
>> > > > Python version as default where not all "regular" image
>> > providers
>> > > are
>> > > > supported in the latest version
>> > > >
>> > > > J.
>> > > >
>> > > > On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com>
>> wrote:
>> > > >
>> > > >> I also started to do more preparation and extracting things that we
>> > will
>> > > >> need to merge first and release providers with some limits lifted.
>> We
>> > > need
>> > > >> to make our providers released in PyPI prepared upfront because
>> > > currently
>> > > >> we are not able to generate PyPI constraints for Python 3.13 - I
>> > already
>> > > >> have one of those merged -
>> > https://github.com/apache/airflow/pull/51994
>> > > >> where we have to exclude `ray` from Python 3.13 and release
>> provider
>> > > before
>> > > >> we get the PR "green" - similarly I have just created
>> > > >> https://github.com/apache/airflow/pull/52060 to all providers that
>> > are
>> > > >> currently limiting pandas (unnecessary it seems from my tests in
>> 3.13
>> > > PR).
>> > > >>
>> > > >> J.
>> > > >>
>> > > >>
>> > > >> On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <
>> amoghdesai.oss@gmail.com
>> > >
>> > > >> wrote:
>> > > >>
>> > > >>> Thanks for taking this on Jarek.
>> > > >>>
>> > > >>> These are not easy tasks to work on and I cannot think of a better
>> > > person
>> > > >>> to do it than you!
>> > > >>>
>> > > >>> I will review it shortly this week.
>> > > >>>
>> > > >>> I think it is a good thing to introduce in Airflow 3.1 and I would
>> > not
>> > > be
>> > > >>> too much in favour
>> > > >>> to cherry pick it to the 3.x series, just for convenience sake.
>> > > >>>
>> > > >>> Thanks & Regards,
>> > > >>> Amogh Desai
>> > > >>>
>> > > >>>
>> > > >>> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com>
>> > wrote:
>> > > >>>
>> > > >>>> Hey here,
>> > > >>>>
>> > > >>>> Over the last few days and weeks I made significant progress with
>> > > Python
>> > > >>>> 3.13 support and while the PR is not yet fully green - we are
>> > getting
>> > > >>>> close. I undrafted my PR recently:
>> > > >>>>
>> > > >>>> https://github.com/apache/airflow/pull/46891
>> > > >>>>
>> > > >>>> I have a kind request for others to start reviewing it, I will be
>> > > happy
>> > > >>> to
>> > > >>>> respond to questions and I am also happy to accept some fixups to
>> > > solve
>> > > >>>> things in a better way if possible, I did a few hacks and left
>> some
>> > > >>> TODOs
>> > > >>>> that might need some creative ideas to be solved in a better,
>> > simpler
>> > > >>> way.
>> > > >>>> I split the PR into several commits - and keep it continuously
>> > rebased
>> > > >>> on
>> > > >>>> top of the latest main. I had to remove some providers (including
>> > FAB
>> > > -
>> > > >>>> which is actually cool, because we can finally see that airflow
>> > > >>> codebase is
>> > > >>>> finally working even without FAB being around) - some of them are
>> > yet
>> > > to
>> > > >>>> implement Python 3.13 (for example Apache Beam is working on it)
>> so
>> > we
>> > > >>>> might add them back soon.
>> > > >>>>
>> > > >>>> Note - Python 3.13 was a WAY more difficult migration than Python
>> > > 3.12 -
>> > > >>>> there were a number of things removed/changed and small
>> behavioral
>> > > >>> changes
>> > > >>>> that we relied on, also the dependencies are catching up way
>> later,
>> > > and
>> > > >>>> there are some tricky dependencies of ours that make things more
>> > > complex
>> > > >>>> when it comes to selecting "min" versions of those. But we are
>> > finally
>> > > >>>> getting there.
>> > > >>>>
>> > > >>>> Looks like we will be on-time to have 3.13 support for Airflow
>> 3.1.
>> > We
>> > > >>>> might also attempt to cherry-pick it for 3.0, but it might be too
>> > much
>> > > >>> of a
>> > > >>>> hassle, so we can decide after we merge this one to main.
>> > > >>>>
>> > > >>>> J.
>> > > >>>>
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > > For additional commands, e-mail: dev-help@airflow.apache.org
>> > >
>> > >
>> >
>>
>
Re: Python 3.13 is coming
Posted by Jarek Potiuk <ja...@potiuk.com>.
Cool. I have been spending more time with my family visiting me, so did not
look yet at the comments - but I will - shortly - and will be happy to
merge it soon :D
On Wed, Jul 16, 2025 at 9:51 AM Aritra Basu <ar...@gmail.com>
wrote:
> Great job on the PR Jarek! Looked through the broad strokes of the PR and
> all looked good to me.
> --
> Regards,
> Aritra Basu
>
> On Wed, 16 Jul 2025, 11:38 am Amogh Desai, <am...@gmail.com>
> wrote:
>
> > Hi Jarek,
> >
> > That was quite a long PR to review. I focused most of my review time on
> the
> > areas
> > I’m more familiar with — (1) and (4). But I ended up going through the
> rest
> > as well 🙂
> >
> > Great job getting that PR green! I really appreciate the inline comments
> > with active TODOS,
> > they are helpful and make it easier to follow the context.
> >
> > Thanks & Regards,
> > Amogh Desai
> >
> >
> > On Wed, Jul 16, 2025 at 1:47 AM Jens Scheffler
> <j_scheffler@gmx.de.invalid
> > >
> > wrote:
> >
> > > Hi Jarek,
> > >
> > > cool! Thanks for all the efforts and the breath in preparing this!
> Added
> > > some comments in PR gut greatly appreciate (1) that notes describe the
> > > upper bindings and (2) that the reference image default stays with 3.12
> > > as 3.13 lags critical providers being ready.
> > >
> > > Jens
> > >
> > > On 15.07.25 19:52, Jarek Potiuk wrote:
> > > > *TL;DR; Python 3.13 is ... ALMOST here... All you need is final
> reviews
> > > and
> > > > merge !!!! *
> > > >
> > > > The PR is finally about to get green - after about 100 or so
> iterations
> > > (or
> > > > so it seems - I opened it on Feb 19th (!) - so it's 5 months in the
> > > > making). https://github.com/apache/airflow/pull/46891. I would
> really
> > > > appreciate some final reviews and approvals.
> > > >
> > > > Jens and Pavan had already made a number of comments over the loooong
> > > > course of this one and I **think** I addressed all of them. I am
> happy
> > to
> > > > get more comments - maybe some of the choices I made are outdated or
> > can
> > > be
> > > > improved - but now having the PR green it will be way easier to
> iterate
> > > on
> > > > it
> > > >
> > > > It took a really long time and a number of iterations and some
> > > opinionated
> > > > choices. Mostly because FAB cannot be installed yet with Python
> 3.13 -
> > > but
> > > > it also gave me an opportunity to move out or run conditionally
> tests -
> > > for
> > > > all cases where FAB was still not completely removed from the core.
> One
> > > of
> > > > our tests (lowest-deps core Python 3.13) runs the complete suite of
> > tests
> > > > for core where `fab-provider` and `flask-app-builder` are completely
> > > > missing from the CI environment (BTW. `uv sync` is an awesome feature
> > of
> > > > `uv` and it made it possible). And a number of changes from other
> > people
> > > > (Jens, Pavan, LIU ZHE YOU, Wei Lee, Vincent, Elad and many others
> had
> > to
> > > > be merged before we could get this one green).
> > > >
> > > > So I have a kind request - for those who know the codebase well -
> > please
> > > > take a look and let me know if you have any comments. Yes, it's a
> > > monstrous
> > > > PR and - unfortunately - it's not easily breakable into smaller ones,
> > > but I
> > > > tried to make it as easy to review as possible. I split it into 4
> > > > independent commits (which will still have to be squashed and merged
> > > > together):
> > > >
> > > > 1) Breeze changes for Python 3.13 - changes in breeze code that
> enable
> > > > Python 3.13
> > > > 2) Pyproject.toml changes in Providers for Python 3.13 (including
> > > > dependencies and python_requires) - big number of files
> > > > 3) README.rst changes for Providers for Python 3.13 (big number of
> > files)
> > > > 4) Actual changes that were necessary to get the PR green for Python
> > 3.13
> > > > (smaller number, but more substantial changes in airflow-core and
> > > provider
> > > > code that makes Python 3.13 work)
> > > >
> > > > Currently excluded providers for Python 3.13:
> > > > ** apache.beam*
> > > > ** apache.kafka*
> > > > ** fab*
> > > > ** **yandex*
> > > > ** ydb*
> > > >
> > > > They are excluded for various reasons - TODO:s left explaining the
> > > reasons
> > > > and we can gradually bring them in as blockers are removed). Detailed
> > > > summary of changes in the PR and copied below.
> > > >
> > > > When you review the code, you can focus on the part that you are
> > > interested
> > > > in (1) - if you want to look at breeze and (4) if you want to look at
> > > code
> > > > changes, and skip rather easily absolutely necessary dependency
> > > requirement
> > > > changes in (2) and mostly-automated changes from (3).
> > > >
> > > > Yep I am asking for a lot, sorry, but I can't see how I could break
> > that
> > > > one down into separate PRs easily.
> > > >
> > > > I have two kind requests to take a closer look specifically:
> > > >
> > > > a) for *Pavan* -> if you could check the aws-wrangler removal
> replacing
> > > it
> > > > with botocore - for docs publishing. I am not 100% sure if I got it
> > > right,
> > > > I did test it but a second pair of hands would be great to test it
> > > >
> > > > b) for *Wei Lee, Kevin Yang, and Rahul* *Vats* (since you were the
> ones
> > > > working on Waeviate in the past) -> I would really appreciate it if
> you
> > > > could test the Weaviate changes I made. Weaviate was one of the
> > > problematic
> > > > changes because dependency update for 3.13 "freed" upgrade to a newer
> > > > version of Waeviate client (>=4.10.0) that used a httpx rather than
> > > > requests - and we had some low-level mock tests for retry scenarios
> > (429,
> > > > 500, ConnectErrors). And in order to fix it, I had to not only have
> > code
> > > in
> > > > tests for mocking either httpx or requests, but I also had to change
> > the
> > > > retriable error detection in Weaviate provider. I **think** I made
> > proper
> > > > changes there, but I have no experience/no Weaviate instance so I had
> > not
> > > > tested it with the "real" weaviate - I updated the tests and mocked
> > httpx
> > > > in the way I **think** it will work, but I am not 100% sure if I got
> it
> > > > right. Actual tests with weaviate would be great (and you had done
> such
> > > > tests before so it might be way easier for you).
> > > >
> > > > J.
> > > >
> > > > -------------
> > > >
> > > > This is the summary of the changes made. Beware .... it's .....
> > > > LOOOONG.....
> > > >
> > > > Add Python 3.13 support for Airflow.
> > > >
> > > > Added Python 3.13 support across the codebase, including:
> > > > * CI/CD workflow updates to handle Python 3.13.
> > > > * Dockerfile and environment variable adjustments for
> > compatibility.
> > > > * Documentation updates to list Python 3.13 as supported.
> > > > * Dependency and code changes for compatibility with new/changed
> > > > libraries (e.g., numpy 2.x, greenlet, pendulum).
> > > > * Test and plugin manager adjustments for Python 3.13-specific
> > > > behaviors.
> > > > * Removal or skipping of tests and features not compatible with
> > > Python
> > > > 3.13.
> > > > * Improved error handling and logging for unsupported
> > > plugins/providers
> > > > on Python 3.13.
> > > > * Updated all provider README.rst files to:
> > > > * List Python 3.13 as a supported version.
> > > > * Adjust dependency requirements for Python 3.13 (e.g.,
> pandas,
> > > > pyarrow, ray, etc.).
> > > > * Add or update notes about excluded or conditionally
> supported
> > > > dependencies for Python 3.13.
> > > > * Document any known incompatibilities or workarounds for
> > specific
> > > > providers.
> > > > * Updated all provider pyproject.toml files and automation to:
> > > > * Add Python 3.13 to the supported classifiers.
> > > > * Adjust requires-python and dependency constraints for Python
> > > 3.13.
> > > > * Exclude Python 3.13 for providers that are not yet
> compatible
> > > (e.g.,
> > > > those depending on Flask AppBuilder).
> > > > * Add conditional dependencies for new versions of libraries
> > > required
> > > > by Python 3.13.
> > > > * Updated Breeze and related scripts to:
> > > > * Build and test with Python 3.13.
> > > > * Update documentation and release scripts to include Python
> > 3.13.
> > > > * Ensure all dev tooling and test environments work with Python
> > > 3.13.
> > > > * Removed remaining FAB related code from core and removed tests
> > for
> > > > older FAB provider versions (<1.3.0)
> > > > * some of the tests in core still depend on FAB - but we make
> them
> > > > optional - both for parsing (we need to skip some imports) and
> > > > execution - because they test a legacy behaviour of embedding
> > > > Flask plugins that is still supported in core.
> > > > * k8s tests now run also with SimpleAuthManager and
> > > SimpleAuthManager
> > > > is used automatically when running tests in Python 3.13
> > > > * migration tests had to be excluded for Python 3.13 because we
> > > cannot
> > > > migrate down on Python 3.13 without FAB tables created. The
> > tests
> > > > will be brought back when FAB is supported or when we release
> > 3.1
> > > > and start working on 3.2, we should be able to migrate down to
> > 3.1
> > > > * Weaviate tests have been updated to handle case where
> > > weaviate-client
> > > > is > 2.10.0 - because it started to use httpx instead of
> > requests,
> > > > and Python 3.13 actually unblocks using older version of
> > weaviate
> > > > (it was previously blocked indirectly via Flask dependencies
> and
> > > > grpcio).
> > > > * Added doc change in our rules on what is the "default" Python
> > > version
> > > > for airflow images - handling the case where we cannot use
> > > "newest"
> > > > Python version as default where not all "regular" image
> > providers
> > > are
> > > > supported in the latest version
> > > >
> > > > J.
> > > >
> > > > On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > >
> > > >> I also started to do more preparation and extracting things that we
> > will
> > > >> need to merge first and release providers with some limits lifted.
> We
> > > need
> > > >> to make our providers released in PyPI prepared upfront because
> > > currently
> > > >> we are not able to generate PyPI constraints for Python 3.13 - I
> > already
> > > >> have one of those merged -
> > https://github.com/apache/airflow/pull/51994
> > > >> where we have to exclude `ray` from Python 3.13 and release provider
> > > before
> > > >> we get the PR "green" - similarly I have just created
> > > >> https://github.com/apache/airflow/pull/52060 to all providers that
> > are
> > > >> currently limiting pandas (unnecessary it seems from my tests in
> 3.13
> > > PR).
> > > >>
> > > >> J.
> > > >>
> > > >>
> > > >> On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <
> amoghdesai.oss@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >>> Thanks for taking this on Jarek.
> > > >>>
> > > >>> These are not easy tasks to work on and I cannot think of a better
> > > person
> > > >>> to do it than you!
> > > >>>
> > > >>> I will review it shortly this week.
> > > >>>
> > > >>> I think it is a good thing to introduce in Airflow 3.1 and I would
> > not
> > > be
> > > >>> too much in favour
> > > >>> to cherry pick it to the 3.x series, just for convenience sake.
> > > >>>
> > > >>> Thanks & Regards,
> > > >>> Amogh Desai
> > > >>>
> > > >>>
> > > >>> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > >>>
> > > >>>> Hey here,
> > > >>>>
> > > >>>> Over the last few days and weeks I made significant progress with
> > > Python
> > > >>>> 3.13 support and while the PR is not yet fully green - we are
> > getting
> > > >>>> close. I undrafted my PR recently:
> > > >>>>
> > > >>>> https://github.com/apache/airflow/pull/46891
> > > >>>>
> > > >>>> I have a kind request for others to start reviewing it, I will be
> > > happy
> > > >>> to
> > > >>>> respond to questions and I am also happy to accept some fixups to
> > > solve
> > > >>>> things in a better way if possible, I did a few hacks and left
> some
> > > >>> TODOs
> > > >>>> that might need some creative ideas to be solved in a better,
> > simpler
> > > >>> way.
> > > >>>> I split the PR into several commits - and keep it continuously
> > rebased
> > > >>> on
> > > >>>> top of the latest main. I had to remove some providers (including
> > FAB
> > > -
> > > >>>> which is actually cool, because we can finally see that airflow
> > > >>> codebase is
> > > >>>> finally working even without FAB being around) - some of them are
> > yet
> > > to
> > > >>>> implement Python 3.13 (for example Apache Beam is working on it)
> so
> > we
> > > >>>> might add them back soon.
> > > >>>>
> > > >>>> Note - Python 3.13 was a WAY more difficult migration than Python
> > > 3.12 -
> > > >>>> there were a number of things removed/changed and small behavioral
> > > >>> changes
> > > >>>> that we relied on, also the dependencies are catching up way
> later,
> > > and
> > > >>>> there are some tricky dependencies of ours that make things more
> > > complex
> > > >>>> when it comes to selecting "min" versions of those. But we are
> > finally
> > > >>>> getting there.
> > > >>>>
> > > >>>> Looks like we will be on-time to have 3.13 support for Airflow
> 3.1.
> > We
> > > >>>> might also attempt to cherry-pick it for 3.0, but it might be too
> > much
> > > >>> of a
> > > >>>> hassle, so we can decide after we merge this one to main.
> > > >>>>
> > > >>>> J.
> > > >>>>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>
Re: Python 3.13 is coming
Posted by Aritra Basu <ar...@gmail.com>.
Great job on the PR Jarek! Looked through the broad strokes of the PR and
all looked good to me.
--
Regards,
Aritra Basu
On Wed, 16 Jul 2025, 11:38 am Amogh Desai, <am...@gmail.com> wrote:
> Hi Jarek,
>
> That was quite a long PR to review. I focused most of my review time on the
> areas
> I’m more familiar with — (1) and (4). But I ended up going through the rest
> as well 🙂
>
> Great job getting that PR green! I really appreciate the inline comments
> with active TODOS,
> they are helpful and make it easier to follow the context.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Wed, Jul 16, 2025 at 1:47 AM Jens Scheffler <j_scheffler@gmx.de.invalid
> >
> wrote:
>
> > Hi Jarek,
> >
> > cool! Thanks for all the efforts and the breath in preparing this! Added
> > some comments in PR gut greatly appreciate (1) that notes describe the
> > upper bindings and (2) that the reference image default stays with 3.12
> > as 3.13 lags critical providers being ready.
> >
> > Jens
> >
> > On 15.07.25 19:52, Jarek Potiuk wrote:
> > > *TL;DR; Python 3.13 is ... ALMOST here... All you need is final reviews
> > and
> > > merge !!!! *
> > >
> > > The PR is finally about to get green - after about 100 or so iterations
> > (or
> > > so it seems - I opened it on Feb 19th (!) - so it's 5 months in the
> > > making). https://github.com/apache/airflow/pull/46891. I would really
> > > appreciate some final reviews and approvals.
> > >
> > > Jens and Pavan had already made a number of comments over the loooong
> > > course of this one and I **think** I addressed all of them. I am happy
> to
> > > get more comments - maybe some of the choices I made are outdated or
> can
> > be
> > > improved - but now having the PR green it will be way easier to iterate
> > on
> > > it
> > >
> > > It took a really long time and a number of iterations and some
> > opinionated
> > > choices. Mostly because FAB cannot be installed yet with Python 3.13 -
> > but
> > > it also gave me an opportunity to move out or run conditionally tests -
> > for
> > > all cases where FAB was still not completely removed from the core. One
> > of
> > > our tests (lowest-deps core Python 3.13) runs the complete suite of
> tests
> > > for core where `fab-provider` and `flask-app-builder` are completely
> > > missing from the CI environment (BTW. `uv sync` is an awesome feature
> of
> > > `uv` and it made it possible). And a number of changes from other
> people
> > > (Jens, Pavan, LIU ZHE YOU, Wei Lee, Vincent, Elad and many others had
> to
> > > be merged before we could get this one green).
> > >
> > > So I have a kind request - for those who know the codebase well -
> please
> > > take a look and let me know if you have any comments. Yes, it's a
> > monstrous
> > > PR and - unfortunately - it's not easily breakable into smaller ones,
> > but I
> > > tried to make it as easy to review as possible. I split it into 4
> > > independent commits (which will still have to be squashed and merged
> > > together):
> > >
> > > 1) Breeze changes for Python 3.13 - changes in breeze code that enable
> > > Python 3.13
> > > 2) Pyproject.toml changes in Providers for Python 3.13 (including
> > > dependencies and python_requires) - big number of files
> > > 3) README.rst changes for Providers for Python 3.13 (big number of
> files)
> > > 4) Actual changes that were necessary to get the PR green for Python
> 3.13
> > > (smaller number, but more substantial changes in airflow-core and
> > provider
> > > code that makes Python 3.13 work)
> > >
> > > Currently excluded providers for Python 3.13:
> > > ** apache.beam*
> > > ** apache.kafka*
> > > ** fab*
> > > ** **yandex*
> > > ** ydb*
> > >
> > > They are excluded for various reasons - TODO:s left explaining the
> > reasons
> > > and we can gradually bring them in as blockers are removed). Detailed
> > > summary of changes in the PR and copied below.
> > >
> > > When you review the code, you can focus on the part that you are
> > interested
> > > in (1) - if you want to look at breeze and (4) if you want to look at
> > code
> > > changes, and skip rather easily absolutely necessary dependency
> > requirement
> > > changes in (2) and mostly-automated changes from (3).
> > >
> > > Yep I am asking for a lot, sorry, but I can't see how I could break
> that
> > > one down into separate PRs easily.
> > >
> > > I have two kind requests to take a closer look specifically:
> > >
> > > a) for *Pavan* -> if you could check the aws-wrangler removal replacing
> > it
> > > with botocore - for docs publishing. I am not 100% sure if I got it
> > right,
> > > I did test it but a second pair of hands would be great to test it
> > >
> > > b) for *Wei Lee, Kevin Yang, and Rahul* *Vats* (since you were the ones
> > > working on Waeviate in the past) -> I would really appreciate it if you
> > > could test the Weaviate changes I made. Weaviate was one of the
> > problematic
> > > changes because dependency update for 3.13 "freed" upgrade to a newer
> > > version of Waeviate client (>=4.10.0) that used a httpx rather than
> > > requests - and we had some low-level mock tests for retry scenarios
> (429,
> > > 500, ConnectErrors). And in order to fix it, I had to not only have
> code
> > in
> > > tests for mocking either httpx or requests, but I also had to change
> the
> > > retriable error detection in Weaviate provider. I **think** I made
> proper
> > > changes there, but I have no experience/no Weaviate instance so I had
> not
> > > tested it with the "real" weaviate - I updated the tests and mocked
> httpx
> > > in the way I **think** it will work, but I am not 100% sure if I got it
> > > right. Actual tests with weaviate would be great (and you had done such
> > > tests before so it might be way easier for you).
> > >
> > > J.
> > >
> > > -------------
> > >
> > > This is the summary of the changes made. Beware .... it's .....
> > > LOOOONG.....
> > >
> > > Add Python 3.13 support for Airflow.
> > >
> > > Added Python 3.13 support across the codebase, including:
> > > * CI/CD workflow updates to handle Python 3.13.
> > > * Dockerfile and environment variable adjustments for
> compatibility.
> > > * Documentation updates to list Python 3.13 as supported.
> > > * Dependency and code changes for compatibility with new/changed
> > > libraries (e.g., numpy 2.x, greenlet, pendulum).
> > > * Test and plugin manager adjustments for Python 3.13-specific
> > > behaviors.
> > > * Removal or skipping of tests and features not compatible with
> > Python
> > > 3.13.
> > > * Improved error handling and logging for unsupported
> > plugins/providers
> > > on Python 3.13.
> > > * Updated all provider README.rst files to:
> > > * List Python 3.13 as a supported version.
> > > * Adjust dependency requirements for Python 3.13 (e.g., pandas,
> > > pyarrow, ray, etc.).
> > > * Add or update notes about excluded or conditionally supported
> > > dependencies for Python 3.13.
> > > * Document any known incompatibilities or workarounds for
> specific
> > > providers.
> > > * Updated all provider pyproject.toml files and automation to:
> > > * Add Python 3.13 to the supported classifiers.
> > > * Adjust requires-python and dependency constraints for Python
> > 3.13.
> > > * Exclude Python 3.13 for providers that are not yet compatible
> > (e.g.,
> > > those depending on Flask AppBuilder).
> > > * Add conditional dependencies for new versions of libraries
> > required
> > > by Python 3.13.
> > > * Updated Breeze and related scripts to:
> > > * Build and test with Python 3.13.
> > > * Update documentation and release scripts to include Python
> 3.13.
> > > * Ensure all dev tooling and test environments work with Python
> > 3.13.
> > > * Removed remaining FAB related code from core and removed tests
> for
> > > older FAB provider versions (<1.3.0)
> > > * some of the tests in core still depend on FAB - but we make them
> > > optional - both for parsing (we need to skip some imports) and
> > > execution - because they test a legacy behaviour of embedding
> > > Flask plugins that is still supported in core.
> > > * k8s tests now run also with SimpleAuthManager and
> > SimpleAuthManager
> > > is used automatically when running tests in Python 3.13
> > > * migration tests had to be excluded for Python 3.13 because we
> > cannot
> > > migrate down on Python 3.13 without FAB tables created. The
> tests
> > > will be brought back when FAB is supported or when we release
> 3.1
> > > and start working on 3.2, we should be able to migrate down to
> 3.1
> > > * Weaviate tests have been updated to handle case where
> > weaviate-client
> > > is > 2.10.0 - because it started to use httpx instead of
> requests,
> > > and Python 3.13 actually unblocks using older version of
> weaviate
> > > (it was previously blocked indirectly via Flask dependencies and
> > > grpcio).
> > > * Added doc change in our rules on what is the "default" Python
> > version
> > > for airflow images - handling the case where we cannot use
> > "newest"
> > > Python version as default where not all "regular" image
> providers
> > are
> > > supported in the latest version
> > >
> > > J.
> > >
> > > On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > >> I also started to do more preparation and extracting things that we
> will
> > >> need to merge first and release providers with some limits lifted. We
> > need
> > >> to make our providers released in PyPI prepared upfront because
> > currently
> > >> we are not able to generate PyPI constraints for Python 3.13 - I
> already
> > >> have one of those merged -
> https://github.com/apache/airflow/pull/51994
> > >> where we have to exclude `ray` from Python 3.13 and release provider
> > before
> > >> we get the PR "green" - similarly I have just created
> > >> https://github.com/apache/airflow/pull/52060 to all providers that
> are
> > >> currently limiting pandas (unnecessary it seems from my tests in 3.13
> > PR).
> > >>
> > >> J.
> > >>
> > >>
> > >> On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <amoghdesai.oss@gmail.com
> >
> > >> wrote:
> > >>
> > >>> Thanks for taking this on Jarek.
> > >>>
> > >>> These are not easy tasks to work on and I cannot think of a better
> > person
> > >>> to do it than you!
> > >>>
> > >>> I will review it shortly this week.
> > >>>
> > >>> I think it is a good thing to introduce in Airflow 3.1 and I would
> not
> > be
> > >>> too much in favour
> > >>> to cherry pick it to the 3.x series, just for convenience sake.
> > >>>
> > >>> Thanks & Regards,
> > >>> Amogh Desai
> > >>>
> > >>>
> > >>> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >>>
> > >>>> Hey here,
> > >>>>
> > >>>> Over the last few days and weeks I made significant progress with
> > Python
> > >>>> 3.13 support and while the PR is not yet fully green - we are
> getting
> > >>>> close. I undrafted my PR recently:
> > >>>>
> > >>>> https://github.com/apache/airflow/pull/46891
> > >>>>
> > >>>> I have a kind request for others to start reviewing it, I will be
> > happy
> > >>> to
> > >>>> respond to questions and I am also happy to accept some fixups to
> > solve
> > >>>> things in a better way if possible, I did a few hacks and left some
> > >>> TODOs
> > >>>> that might need some creative ideas to be solved in a better,
> simpler
> > >>> way.
> > >>>> I split the PR into several commits - and keep it continuously
> rebased
> > >>> on
> > >>>> top of the latest main. I had to remove some providers (including
> FAB
> > -
> > >>>> which is actually cool, because we can finally see that airflow
> > >>> codebase is
> > >>>> finally working even without FAB being around) - some of them are
> yet
> > to
> > >>>> implement Python 3.13 (for example Apache Beam is working on it) so
> we
> > >>>> might add them back soon.
> > >>>>
> > >>>> Note - Python 3.13 was a WAY more difficult migration than Python
> > 3.12 -
> > >>>> there were a number of things removed/changed and small behavioral
> > >>> changes
> > >>>> that we relied on, also the dependencies are catching up way later,
> > and
> > >>>> there are some tricky dependencies of ours that make things more
> > complex
> > >>>> when it comes to selecting "min" versions of those. But we are
> finally
> > >>>> getting there.
> > >>>>
> > >>>> Looks like we will be on-time to have 3.13 support for Airflow 3.1.
> We
> > >>>> might also attempt to cherry-pick it for 3.0, but it might be too
> much
> > >>> of a
> > >>>> hassle, so we can decide after we merge this one to main.
> > >>>>
> > >>>> J.
> > >>>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: Python 3.13 is coming
Posted by Amogh Desai <am...@gmail.com>.
Hi Jarek,
That was quite a long PR to review. I focused most of my review time on the
areas
I’m more familiar with — (1) and (4). But I ended up going through the rest
as well 🙂
Great job getting that PR green! I really appreciate the inline comments
with active TODOS,
they are helpful and make it easier to follow the context.
Thanks & Regards,
Amogh Desai
On Wed, Jul 16, 2025 at 1:47 AM Jens Scheffler <j_...@gmx.de.invalid>
wrote:
> Hi Jarek,
>
> cool! Thanks for all the efforts and the breath in preparing this! Added
> some comments in PR gut greatly appreciate (1) that notes describe the
> upper bindings and (2) that the reference image default stays with 3.12
> as 3.13 lags critical providers being ready.
>
> Jens
>
> On 15.07.25 19:52, Jarek Potiuk wrote:
> > *TL;DR; Python 3.13 is ... ALMOST here... All you need is final reviews
> and
> > merge !!!! *
> >
> > The PR is finally about to get green - after about 100 or so iterations
> (or
> > so it seems - I opened it on Feb 19th (!) - so it's 5 months in the
> > making). https://github.com/apache/airflow/pull/46891. I would really
> > appreciate some final reviews and approvals.
> >
> > Jens and Pavan had already made a number of comments over the loooong
> > course of this one and I **think** I addressed all of them. I am happy to
> > get more comments - maybe some of the choices I made are outdated or can
> be
> > improved - but now having the PR green it will be way easier to iterate
> on
> > it
> >
> > It took a really long time and a number of iterations and some
> opinionated
> > choices. Mostly because FAB cannot be installed yet with Python 3.13 -
> but
> > it also gave me an opportunity to move out or run conditionally tests -
> for
> > all cases where FAB was still not completely removed from the core. One
> of
> > our tests (lowest-deps core Python 3.13) runs the complete suite of tests
> > for core where `fab-provider` and `flask-app-builder` are completely
> > missing from the CI environment (BTW. `uv sync` is an awesome feature of
> > `uv` and it made it possible). And a number of changes from other people
> > (Jens, Pavan, LIU ZHE YOU, Wei Lee, Vincent, Elad and many others had to
> > be merged before we could get this one green).
> >
> > So I have a kind request - for those who know the codebase well - please
> > take a look and let me know if you have any comments. Yes, it's a
> monstrous
> > PR and - unfortunately - it's not easily breakable into smaller ones,
> but I
> > tried to make it as easy to review as possible. I split it into 4
> > independent commits (which will still have to be squashed and merged
> > together):
> >
> > 1) Breeze changes for Python 3.13 - changes in breeze code that enable
> > Python 3.13
> > 2) Pyproject.toml changes in Providers for Python 3.13 (including
> > dependencies and python_requires) - big number of files
> > 3) README.rst changes for Providers for Python 3.13 (big number of files)
> > 4) Actual changes that were necessary to get the PR green for Python 3.13
> > (smaller number, but more substantial changes in airflow-core and
> provider
> > code that makes Python 3.13 work)
> >
> > Currently excluded providers for Python 3.13:
> > ** apache.beam*
> > ** apache.kafka*
> > ** fab*
> > ** **yandex*
> > ** ydb*
> >
> > They are excluded for various reasons - TODO:s left explaining the
> reasons
> > and we can gradually bring them in as blockers are removed). Detailed
> > summary of changes in the PR and copied below.
> >
> > When you review the code, you can focus on the part that you are
> interested
> > in (1) - if you want to look at breeze and (4) if you want to look at
> code
> > changes, and skip rather easily absolutely necessary dependency
> requirement
> > changes in (2) and mostly-automated changes from (3).
> >
> > Yep I am asking for a lot, sorry, but I can't see how I could break that
> > one down into separate PRs easily.
> >
> > I have two kind requests to take a closer look specifically:
> >
> > a) for *Pavan* -> if you could check the aws-wrangler removal replacing
> it
> > with botocore - for docs publishing. I am not 100% sure if I got it
> right,
> > I did test it but a second pair of hands would be great to test it
> >
> > b) for *Wei Lee, Kevin Yang, and Rahul* *Vats* (since you were the ones
> > working on Waeviate in the past) -> I would really appreciate it if you
> > could test the Weaviate changes I made. Weaviate was one of the
> problematic
> > changes because dependency update for 3.13 "freed" upgrade to a newer
> > version of Waeviate client (>=4.10.0) that used a httpx rather than
> > requests - and we had some low-level mock tests for retry scenarios (429,
> > 500, ConnectErrors). And in order to fix it, I had to not only have code
> in
> > tests for mocking either httpx or requests, but I also had to change the
> > retriable error detection in Weaviate provider. I **think** I made proper
> > changes there, but I have no experience/no Weaviate instance so I had not
> > tested it with the "real" weaviate - I updated the tests and mocked httpx
> > in the way I **think** it will work, but I am not 100% sure if I got it
> > right. Actual tests with weaviate would be great (and you had done such
> > tests before so it might be way easier for you).
> >
> > J.
> >
> > -------------
> >
> > This is the summary of the changes made. Beware .... it's .....
> > LOOOONG.....
> >
> > Add Python 3.13 support for Airflow.
> >
> > Added Python 3.13 support across the codebase, including:
> > * CI/CD workflow updates to handle Python 3.13.
> > * Dockerfile and environment variable adjustments for compatibility.
> > * Documentation updates to list Python 3.13 as supported.
> > * Dependency and code changes for compatibility with new/changed
> > libraries (e.g., numpy 2.x, greenlet, pendulum).
> > * Test and plugin manager adjustments for Python 3.13-specific
> > behaviors.
> > * Removal or skipping of tests and features not compatible with
> Python
> > 3.13.
> > * Improved error handling and logging for unsupported
> plugins/providers
> > on Python 3.13.
> > * Updated all provider README.rst files to:
> > * List Python 3.13 as a supported version.
> > * Adjust dependency requirements for Python 3.13 (e.g., pandas,
> > pyarrow, ray, etc.).
> > * Add or update notes about excluded or conditionally supported
> > dependencies for Python 3.13.
> > * Document any known incompatibilities or workarounds for specific
> > providers.
> > * Updated all provider pyproject.toml files and automation to:
> > * Add Python 3.13 to the supported classifiers.
> > * Adjust requires-python and dependency constraints for Python
> 3.13.
> > * Exclude Python 3.13 for providers that are not yet compatible
> (e.g.,
> > those depending on Flask AppBuilder).
> > * Add conditional dependencies for new versions of libraries
> required
> > by Python 3.13.
> > * Updated Breeze and related scripts to:
> > * Build and test with Python 3.13.
> > * Update documentation and release scripts to include Python 3.13.
> > * Ensure all dev tooling and test environments work with Python
> 3.13.
> > * Removed remaining FAB related code from core and removed tests for
> > older FAB provider versions (<1.3.0)
> > * some of the tests in core still depend on FAB - but we make them
> > optional - both for parsing (we need to skip some imports) and
> > execution - because they test a legacy behaviour of embedding
> > Flask plugins that is still supported in core.
> > * k8s tests now run also with SimpleAuthManager and
> SimpleAuthManager
> > is used automatically when running tests in Python 3.13
> > * migration tests had to be excluded for Python 3.13 because we
> cannot
> > migrate down on Python 3.13 without FAB tables created. The tests
> > will be brought back when FAB is supported or when we release 3.1
> > and start working on 3.2, we should be able to migrate down to 3.1
> > * Weaviate tests have been updated to handle case where
> weaviate-client
> > is > 2.10.0 - because it started to use httpx instead of requests,
> > and Python 3.13 actually unblocks using older version of weaviate
> > (it was previously blocked indirectly via Flask dependencies and
> > grpcio).
> > * Added doc change in our rules on what is the "default" Python
> version
> > for airflow images - handling the case where we cannot use
> "newest"
> > Python version as default where not all "regular" image providers
> are
> > supported in the latest version
> >
> > J.
> >
> > On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >> I also started to do more preparation and extracting things that we will
> >> need to merge first and release providers with some limits lifted. We
> need
> >> to make our providers released in PyPI prepared upfront because
> currently
> >> we are not able to generate PyPI constraints for Python 3.13 - I already
> >> have one of those merged - https://github.com/apache/airflow/pull/51994
> >> where we have to exclude `ray` from Python 3.13 and release provider
> before
> >> we get the PR "green" - similarly I have just created
> >> https://github.com/apache/airflow/pull/52060 to all providers that are
> >> currently limiting pandas (unnecessary it seems from my tests in 3.13
> PR).
> >>
> >> J.
> >>
> >>
> >> On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <am...@gmail.com>
> >> wrote:
> >>
> >>> Thanks for taking this on Jarek.
> >>>
> >>> These are not easy tasks to work on and I cannot think of a better
> person
> >>> to do it than you!
> >>>
> >>> I will review it shortly this week.
> >>>
> >>> I think it is a good thing to introduce in Airflow 3.1 and I would not
> be
> >>> too much in favour
> >>> to cherry pick it to the 3.x series, just for convenience sake.
> >>>
> >>> Thanks & Regards,
> >>> Amogh Desai
> >>>
> >>>
> >>> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>>
> >>>> Hey here,
> >>>>
> >>>> Over the last few days and weeks I made significant progress with
> Python
> >>>> 3.13 support and while the PR is not yet fully green - we are getting
> >>>> close. I undrafted my PR recently:
> >>>>
> >>>> https://github.com/apache/airflow/pull/46891
> >>>>
> >>>> I have a kind request for others to start reviewing it, I will be
> happy
> >>> to
> >>>> respond to questions and I am also happy to accept some fixups to
> solve
> >>>> things in a better way if possible, I did a few hacks and left some
> >>> TODOs
> >>>> that might need some creative ideas to be solved in a better, simpler
> >>> way.
> >>>> I split the PR into several commits - and keep it continuously rebased
> >>> on
> >>>> top of the latest main. I had to remove some providers (including FAB
> -
> >>>> which is actually cool, because we can finally see that airflow
> >>> codebase is
> >>>> finally working even without FAB being around) - some of them are yet
> to
> >>>> implement Python 3.13 (for example Apache Beam is working on it) so we
> >>>> might add them back soon.
> >>>>
> >>>> Note - Python 3.13 was a WAY more difficult migration than Python
> 3.12 -
> >>>> there were a number of things removed/changed and small behavioral
> >>> changes
> >>>> that we relied on, also the dependencies are catching up way later,
> and
> >>>> there are some tricky dependencies of ours that make things more
> complex
> >>>> when it comes to selecting "min" versions of those. But we are finally
> >>>> getting there.
> >>>>
> >>>> Looks like we will be on-time to have 3.13 support for Airflow 3.1. We
> >>>> might also attempt to cherry-pick it for 3.0, but it might be too much
> >>> of a
> >>>> hassle, so we can decide after we merge this one to main.
> >>>>
> >>>> J.
> >>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: Python 3.13 is coming
Posted by Jens Scheffler <j_...@gmx.de.INVALID>.
Hi Jarek,
cool! Thanks for all the efforts and the breath in preparing this! Added
some comments in PR gut greatly appreciate (1) that notes describe the
upper bindings and (2) that the reference image default stays with 3.12
as 3.13 lags critical providers being ready.
Jens
On 15.07.25 19:52, Jarek Potiuk wrote:
> *TL;DR; Python 3.13 is ... ALMOST here... All you need is final reviews and
> merge !!!! *
>
> The PR is finally about to get green - after about 100 or so iterations (or
> so it seems - I opened it on Feb 19th (!) - so it's 5 months in the
> making). https://github.com/apache/airflow/pull/46891. I would really
> appreciate some final reviews and approvals.
>
> Jens and Pavan had already made a number of comments over the loooong
> course of this one and I **think** I addressed all of them. I am happy to
> get more comments - maybe some of the choices I made are outdated or can be
> improved - but now having the PR green it will be way easier to iterate on
> it
>
> It took a really long time and a number of iterations and some opinionated
> choices. Mostly because FAB cannot be installed yet with Python 3.13 - but
> it also gave me an opportunity to move out or run conditionally tests - for
> all cases where FAB was still not completely removed from the core. One of
> our tests (lowest-deps core Python 3.13) runs the complete suite of tests
> for core where `fab-provider` and `flask-app-builder` are completely
> missing from the CI environment (BTW. `uv sync` is an awesome feature of
> `uv` and it made it possible). And a number of changes from other people
> (Jens, Pavan, LIU ZHE YOU, Wei Lee, Vincent, Elad and many others had to
> be merged before we could get this one green).
>
> So I have a kind request - for those who know the codebase well - please
> take a look and let me know if you have any comments. Yes, it's a monstrous
> PR and - unfortunately - it's not easily breakable into smaller ones, but I
> tried to make it as easy to review as possible. I split it into 4
> independent commits (which will still have to be squashed and merged
> together):
>
> 1) Breeze changes for Python 3.13 - changes in breeze code that enable
> Python 3.13
> 2) Pyproject.toml changes in Providers for Python 3.13 (including
> dependencies and python_requires) - big number of files
> 3) README.rst changes for Providers for Python 3.13 (big number of files)
> 4) Actual changes that were necessary to get the PR green for Python 3.13
> (smaller number, but more substantial changes in airflow-core and provider
> code that makes Python 3.13 work)
>
> Currently excluded providers for Python 3.13:
> ** apache.beam*
> ** apache.kafka*
> ** fab*
> ** **yandex*
> ** ydb*
>
> They are excluded for various reasons - TODO:s left explaining the reasons
> and we can gradually bring them in as blockers are removed). Detailed
> summary of changes in the PR and copied below.
>
> When you review the code, you can focus on the part that you are interested
> in (1) - if you want to look at breeze and (4) if you want to look at code
> changes, and skip rather easily absolutely necessary dependency requirement
> changes in (2) and mostly-automated changes from (3).
>
> Yep I am asking for a lot, sorry, but I can't see how I could break that
> one down into separate PRs easily.
>
> I have two kind requests to take a closer look specifically:
>
> a) for *Pavan* -> if you could check the aws-wrangler removal replacing it
> with botocore - for docs publishing. I am not 100% sure if I got it right,
> I did test it but a second pair of hands would be great to test it
>
> b) for *Wei Lee, Kevin Yang, and Rahul* *Vats* (since you were the ones
> working on Waeviate in the past) -> I would really appreciate it if you
> could test the Weaviate changes I made. Weaviate was one of the problematic
> changes because dependency update for 3.13 "freed" upgrade to a newer
> version of Waeviate client (>=4.10.0) that used a httpx rather than
> requests - and we had some low-level mock tests for retry scenarios (429,
> 500, ConnectErrors). And in order to fix it, I had to not only have code in
> tests for mocking either httpx or requests, but I also had to change the
> retriable error detection in Weaviate provider. I **think** I made proper
> changes there, but I have no experience/no Weaviate instance so I had not
> tested it with the "real" weaviate - I updated the tests and mocked httpx
> in the way I **think** it will work, but I am not 100% sure if I got it
> right. Actual tests with weaviate would be great (and you had done such
> tests before so it might be way easier for you).
>
> J.
>
> -------------
>
> This is the summary of the changes made. Beware .... it's .....
> LOOOONG.....
>
> Add Python 3.13 support for Airflow.
>
> Added Python 3.13 support across the codebase, including:
> * CI/CD workflow updates to handle Python 3.13.
> * Dockerfile and environment variable adjustments for compatibility.
> * Documentation updates to list Python 3.13 as supported.
> * Dependency and code changes for compatibility with new/changed
> libraries (e.g., numpy 2.x, greenlet, pendulum).
> * Test and plugin manager adjustments for Python 3.13-specific
> behaviors.
> * Removal or skipping of tests and features not compatible with Python
> 3.13.
> * Improved error handling and logging for unsupported plugins/providers
> on Python 3.13.
> * Updated all provider README.rst files to:
> * List Python 3.13 as a supported version.
> * Adjust dependency requirements for Python 3.13 (e.g., pandas,
> pyarrow, ray, etc.).
> * Add or update notes about excluded or conditionally supported
> dependencies for Python 3.13.
> * Document any known incompatibilities or workarounds for specific
> providers.
> * Updated all provider pyproject.toml files and automation to:
> * Add Python 3.13 to the supported classifiers.
> * Adjust requires-python and dependency constraints for Python 3.13.
> * Exclude Python 3.13 for providers that are not yet compatible (e.g.,
> those depending on Flask AppBuilder).
> * Add conditional dependencies for new versions of libraries required
> by Python 3.13.
> * Updated Breeze and related scripts to:
> * Build and test with Python 3.13.
> * Update documentation and release scripts to include Python 3.13.
> * Ensure all dev tooling and test environments work with Python 3.13.
> * Removed remaining FAB related code from core and removed tests for
> older FAB provider versions (<1.3.0)
> * some of the tests in core still depend on FAB - but we make them
> optional - both for parsing (we need to skip some imports) and
> execution - because they test a legacy behaviour of embedding
> Flask plugins that is still supported in core.
> * k8s tests now run also with SimpleAuthManager and SimpleAuthManager
> is used automatically when running tests in Python 3.13
> * migration tests had to be excluded for Python 3.13 because we cannot
> migrate down on Python 3.13 without FAB tables created. The tests
> will be brought back when FAB is supported or when we release 3.1
> and start working on 3.2, we should be able to migrate down to 3.1
> * Weaviate tests have been updated to handle case where weaviate-client
> is > 2.10.0 - because it started to use httpx instead of requests,
> and Python 3.13 actually unblocks using older version of weaviate
> (it was previously blocked indirectly via Flask dependencies and
> grpcio).
> * Added doc change in our rules on what is the "default" Python version
> for airflow images - handling the case where we cannot use "newest"
> Python version as default where not all "regular" image providers are
> supported in the latest version
>
> J.
>
> On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> I also started to do more preparation and extracting things that we will
>> need to merge first and release providers with some limits lifted. We need
>> to make our providers released in PyPI prepared upfront because currently
>> we are not able to generate PyPI constraints for Python 3.13 - I already
>> have one of those merged - https://github.com/apache/airflow/pull/51994
>> where we have to exclude `ray` from Python 3.13 and release provider before
>> we get the PR "green" - similarly I have just created
>> https://github.com/apache/airflow/pull/52060 to all providers that are
>> currently limiting pandas (unnecessary it seems from my tests in 3.13 PR).
>>
>> J.
>>
>>
>> On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <am...@gmail.com>
>> wrote:
>>
>>> Thanks for taking this on Jarek.
>>>
>>> These are not easy tasks to work on and I cannot think of a better person
>>> to do it than you!
>>>
>>> I will review it shortly this week.
>>>
>>> I think it is a good thing to introduce in Airflow 3.1 and I would not be
>>> too much in favour
>>> to cherry pick it to the 3.x series, just for convenience sake.
>>>
>>> Thanks & Regards,
>>> Amogh Desai
>>>
>>>
>>> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>>
>>>> Hey here,
>>>>
>>>> Over the last few days and weeks I made significant progress with Python
>>>> 3.13 support and while the PR is not yet fully green - we are getting
>>>> close. I undrafted my PR recently:
>>>>
>>>> https://github.com/apache/airflow/pull/46891
>>>>
>>>> I have a kind request for others to start reviewing it, I will be happy
>>> to
>>>> respond to questions and I am also happy to accept some fixups to solve
>>>> things in a better way if possible, I did a few hacks and left some
>>> TODOs
>>>> that might need some creative ideas to be solved in a better, simpler
>>> way.
>>>> I split the PR into several commits - and keep it continuously rebased
>>> on
>>>> top of the latest main. I had to remove some providers (including FAB -
>>>> which is actually cool, because we can finally see that airflow
>>> codebase is
>>>> finally working even without FAB being around) - some of them are yet to
>>>> implement Python 3.13 (for example Apache Beam is working on it) so we
>>>> might add them back soon.
>>>>
>>>> Note - Python 3.13 was a WAY more difficult migration than Python 3.12 -
>>>> there were a number of things removed/changed and small behavioral
>>> changes
>>>> that we relied on, also the dependencies are catching up way later, and
>>>> there are some tricky dependencies of ours that make things more complex
>>>> when it comes to selecting "min" versions of those. But we are finally
>>>> getting there.
>>>>
>>>> Looks like we will be on-time to have 3.13 support for Airflow 3.1. We
>>>> might also attempt to cherry-pick it for 3.0, but it might be too much
>>> of a
>>>> hassle, so we can decide after we merge this one to main.
>>>>
>>>> J.
>>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: Python 3.13 is coming
Posted by Jarek Potiuk <ja...@potiuk.com>.
*TL;DR; Python 3.13 is ... ALMOST here... All you need is final reviews and
merge !!!! *
The PR is finally about to get green - after about 100 or so iterations (or
so it seems - I opened it on Feb 19th (!) - so it's 5 months in the
making). https://github.com/apache/airflow/pull/46891. I would really
appreciate some final reviews and approvals.
Jens and Pavan had already made a number of comments over the loooong
course of this one and I **think** I addressed all of them. I am happy to
get more comments - maybe some of the choices I made are outdated or can be
improved - but now having the PR green it will be way easier to iterate on
it
It took a really long time and a number of iterations and some opinionated
choices. Mostly because FAB cannot be installed yet with Python 3.13 - but
it also gave me an opportunity to move out or run conditionally tests - for
all cases where FAB was still not completely removed from the core. One of
our tests (lowest-deps core Python 3.13) runs the complete suite of tests
for core where `fab-provider` and `flask-app-builder` are completely
missing from the CI environment (BTW. `uv sync` is an awesome feature of
`uv` and it made it possible). And a number of changes from other people
(Jens, Pavan, LIU ZHE YOU, Wei Lee, Vincent, Elad and many others had to
be merged before we could get this one green).
So I have a kind request - for those who know the codebase well - please
take a look and let me know if you have any comments. Yes, it's a monstrous
PR and - unfortunately - it's not easily breakable into smaller ones, but I
tried to make it as easy to review as possible. I split it into 4
independent commits (which will still have to be squashed and merged
together):
1) Breeze changes for Python 3.13 - changes in breeze code that enable
Python 3.13
2) Pyproject.toml changes in Providers for Python 3.13 (including
dependencies and python_requires) - big number of files
3) README.rst changes for Providers for Python 3.13 (big number of files)
4) Actual changes that were necessary to get the PR green for Python 3.13
(smaller number, but more substantial changes in airflow-core and provider
code that makes Python 3.13 work)
Currently excluded providers for Python 3.13:
** apache.beam*
** apache.kafka*
** fab*
** **yandex*
** ydb*
They are excluded for various reasons - TODO:s left explaining the reasons
and we can gradually bring them in as blockers are removed). Detailed
summary of changes in the PR and copied below.
When you review the code, you can focus on the part that you are interested
in (1) - if you want to look at breeze and (4) if you want to look at code
changes, and skip rather easily absolutely necessary dependency requirement
changes in (2) and mostly-automated changes from (3).
Yep I am asking for a lot, sorry, but I can't see how I could break that
one down into separate PRs easily.
I have two kind requests to take a closer look specifically:
a) for *Pavan* -> if you could check the aws-wrangler removal replacing it
with botocore - for docs publishing. I am not 100% sure if I got it right,
I did test it but a second pair of hands would be great to test it
b) for *Wei Lee, Kevin Yang, and Rahul* *Vats* (since you were the ones
working on Waeviate in the past) -> I would really appreciate it if you
could test the Weaviate changes I made. Weaviate was one of the problematic
changes because dependency update for 3.13 "freed" upgrade to a newer
version of Waeviate client (>=4.10.0) that used a httpx rather than
requests - and we had some low-level mock tests for retry scenarios (429,
500, ConnectErrors). And in order to fix it, I had to not only have code in
tests for mocking either httpx or requests, but I also had to change the
retriable error detection in Weaviate provider. I **think** I made proper
changes there, but I have no experience/no Weaviate instance so I had not
tested it with the "real" weaviate - I updated the tests and mocked httpx
in the way I **think** it will work, but I am not 100% sure if I got it
right. Actual tests with weaviate would be great (and you had done such
tests before so it might be way easier for you).
J.
-------------
This is the summary of the changes made. Beware .... it's .....
LOOOONG.....
Add Python 3.13 support for Airflow.
Added Python 3.13 support across the codebase, including:
* CI/CD workflow updates to handle Python 3.13.
* Dockerfile and environment variable adjustments for compatibility.
* Documentation updates to list Python 3.13 as supported.
* Dependency and code changes for compatibility with new/changed
libraries (e.g., numpy 2.x, greenlet, pendulum).
* Test and plugin manager adjustments for Python 3.13-specific
behaviors.
* Removal or skipping of tests and features not compatible with Python
3.13.
* Improved error handling and logging for unsupported plugins/providers
on Python 3.13.
* Updated all provider README.rst files to:
* List Python 3.13 as a supported version.
* Adjust dependency requirements for Python 3.13 (e.g., pandas,
pyarrow, ray, etc.).
* Add or update notes about excluded or conditionally supported
dependencies for Python 3.13.
* Document any known incompatibilities or workarounds for specific
providers.
* Updated all provider pyproject.toml files and automation to:
* Add Python 3.13 to the supported classifiers.
* Adjust requires-python and dependency constraints for Python 3.13.
* Exclude Python 3.13 for providers that are not yet compatible (e.g.,
those depending on Flask AppBuilder).
* Add conditional dependencies for new versions of libraries required
by Python 3.13.
* Updated Breeze and related scripts to:
* Build and test with Python 3.13.
* Update documentation and release scripts to include Python 3.13.
* Ensure all dev tooling and test environments work with Python 3.13.
* Removed remaining FAB related code from core and removed tests for
older FAB provider versions (<1.3.0)
* some of the tests in core still depend on FAB - but we make them
optional - both for parsing (we need to skip some imports) and
execution - because they test a legacy behaviour of embedding
Flask plugins that is still supported in core.
* k8s tests now run also with SimpleAuthManager and SimpleAuthManager
is used automatically when running tests in Python 3.13
* migration tests had to be excluded for Python 3.13 because we cannot
migrate down on Python 3.13 without FAB tables created. The tests
will be brought back when FAB is supported or when we release 3.1
and start working on 3.2, we should be able to migrate down to 3.1
* Weaviate tests have been updated to handle case where weaviate-client
is > 2.10.0 - because it started to use httpx instead of requests,
and Python 3.13 actually unblocks using older version of weaviate
(it was previously blocked indirectly via Flask dependencies and
grpcio).
* Added doc change in our rules on what is the "default" Python version
for airflow images - handling the case where we cannot use "newest"
Python version as default where not all "regular" image providers are
supported in the latest version
J.
On Mon, Jun 23, 2025 at 8:00 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> I also started to do more preparation and extracting things that we will
> need to merge first and release providers with some limits lifted. We need
> to make our providers released in PyPI prepared upfront because currently
> we are not able to generate PyPI constraints for Python 3.13 - I already
> have one of those merged - https://github.com/apache/airflow/pull/51994
> where we have to exclude `ray` from Python 3.13 and release provider before
> we get the PR "green" - similarly I have just created
> https://github.com/apache/airflow/pull/52060 to all providers that are
> currently limiting pandas (unnecessary it seems from my tests in 3.13 PR).
>
> J.
>
>
> On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <am...@gmail.com>
> wrote:
>
>> Thanks for taking this on Jarek.
>>
>> These are not easy tasks to work on and I cannot think of a better person
>> to do it than you!
>>
>> I will review it shortly this week.
>>
>> I think it is a good thing to introduce in Airflow 3.1 and I would not be
>> too much in favour
>> to cherry pick it to the 3.x series, just for convenience sake.
>>
>> Thanks & Regards,
>> Amogh Desai
>>
>>
>> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>> > Hey here,
>> >
>> > Over the last few days and weeks I made significant progress with Python
>> > 3.13 support and while the PR is not yet fully green - we are getting
>> > close. I undrafted my PR recently:
>> >
>> > https://github.com/apache/airflow/pull/46891
>> >
>> > I have a kind request for others to start reviewing it, I will be happy
>> to
>> > respond to questions and I am also happy to accept some fixups to solve
>> > things in a better way if possible, I did a few hacks and left some
>> TODOs
>> > that might need some creative ideas to be solved in a better, simpler
>> way.
>> >
>> > I split the PR into several commits - and keep it continuously rebased
>> on
>> > top of the latest main. I had to remove some providers (including FAB -
>> > which is actually cool, because we can finally see that airflow
>> codebase is
>> > finally working even without FAB being around) - some of them are yet to
>> > implement Python 3.13 (for example Apache Beam is working on it) so we
>> > might add them back soon.
>> >
>> > Note - Python 3.13 was a WAY more difficult migration than Python 3.12 -
>> > there were a number of things removed/changed and small behavioral
>> changes
>> > that we relied on, also the dependencies are catching up way later, and
>> > there are some tricky dependencies of ours that make things more complex
>> > when it comes to selecting "min" versions of those. But we are finally
>> > getting there.
>> >
>> > Looks like we will be on-time to have 3.13 support for Airflow 3.1. We
>> > might also attempt to cherry-pick it for 3.0, but it might be too much
>> of a
>> > hassle, so we can decide after we merge this one to main.
>> >
>> > J.
>> >
>>
>
Re: Python 3.13 is coming
Posted by Jarek Potiuk <ja...@potiuk.com>.
I also started to do more preparation and extracting things that we will
need to merge first and release providers with some limits lifted. We need
to make our providers released in PyPI prepared upfront because currently
we are not able to generate PyPI constraints for Python 3.13 - I already
have one of those merged - https://github.com/apache/airflow/pull/51994
where we have to exclude `ray` from Python 3.13 and release provider before
we get the PR "green" - similarly I have just created
https://github.com/apache/airflow/pull/52060 to all providers that are
currently limiting pandas (unnecessary it seems from my tests in 3.13 PR).
J.
On Mon, Jun 23, 2025 at 7:46 AM Amogh Desai <am...@gmail.com>
wrote:
> Thanks for taking this on Jarek.
>
> These are not easy tasks to work on and I cannot think of a better person
> to do it than you!
>
> I will review it shortly this week.
>
> I think it is a good thing to introduce in Airflow 3.1 and I would not be
> too much in favour
> to cherry pick it to the 3.x series, just for convenience sake.
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > Hey here,
> >
> > Over the last few days and weeks I made significant progress with Python
> > 3.13 support and while the PR is not yet fully green - we are getting
> > close. I undrafted my PR recently:
> >
> > https://github.com/apache/airflow/pull/46891
> >
> > I have a kind request for others to start reviewing it, I will be happy
> to
> > respond to questions and I am also happy to accept some fixups to solve
> > things in a better way if possible, I did a few hacks and left some TODOs
> > that might need some creative ideas to be solved in a better, simpler
> way.
> >
> > I split the PR into several commits - and keep it continuously rebased on
> > top of the latest main. I had to remove some providers (including FAB -
> > which is actually cool, because we can finally see that airflow codebase
> is
> > finally working even without FAB being around) - some of them are yet to
> > implement Python 3.13 (for example Apache Beam is working on it) so we
> > might add them back soon.
> >
> > Note - Python 3.13 was a WAY more difficult migration than Python 3.12 -
> > there were a number of things removed/changed and small behavioral
> changes
> > that we relied on, also the dependencies are catching up way later, and
> > there are some tricky dependencies of ours that make things more complex
> > when it comes to selecting "min" versions of those. But we are finally
> > getting there.
> >
> > Looks like we will be on-time to have 3.13 support for Airflow 3.1. We
> > might also attempt to cherry-pick it for 3.0, but it might be too much
> of a
> > hassle, so we can decide after we merge this one to main.
> >
> > J.
> >
>
Re: Python 3.13 is coming
Posted by Amogh Desai <am...@gmail.com>.
Thanks for taking this on Jarek.
These are not easy tasks to work on and I cannot think of a better person
to do it than you!
I will review it shortly this week.
I think it is a good thing to introduce in Airflow 3.1 and I would not be
too much in favour
to cherry pick it to the 3.x series, just for convenience sake.
Thanks & Regards,
Amogh Desai
On Fri, Jun 20, 2025 at 2:47 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hey here,
>
> Over the last few days and weeks I made significant progress with Python
> 3.13 support and while the PR is not yet fully green - we are getting
> close. I undrafted my PR recently:
>
> https://github.com/apache/airflow/pull/46891
>
> I have a kind request for others to start reviewing it, I will be happy to
> respond to questions and I am also happy to accept some fixups to solve
> things in a better way if possible, I did a few hacks and left some TODOs
> that might need some creative ideas to be solved in a better, simpler way.
>
> I split the PR into several commits - and keep it continuously rebased on
> top of the latest main. I had to remove some providers (including FAB -
> which is actually cool, because we can finally see that airflow codebase is
> finally working even without FAB being around) - some of them are yet to
> implement Python 3.13 (for example Apache Beam is working on it) so we
> might add them back soon.
>
> Note - Python 3.13 was a WAY more difficult migration than Python 3.12 -
> there were a number of things removed/changed and small behavioral changes
> that we relied on, also the dependencies are catching up way later, and
> there are some tricky dependencies of ours that make things more complex
> when it comes to selecting "min" versions of those. But we are finally
> getting there.
>
> Looks like we will be on-time to have 3.13 support for Airflow 3.1. We
> might also attempt to cherry-pick it for 3.0, but it might be too much of a
> hassle, so we can decide after we merge this one to main.
>
> J.
>