You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Abhishek Bhakat <ab...@astronomer.io.INVALID> on 2025/05/07 14:48:28 UTC
[DISCUSS] Migrating to SQLAlchemy 2
Hi All,
I've hit a little deadlock.
We want to use SQLAlchemy 2.x ORM typing (Mapped[...] =
mapped_column(...)) with
PEP 604 union types (int | None) in model annotations and must support
Python 3.9 (which does not support int | None at runtime).
We do not want to use Optional[...] (ruff UP007 is enforced). And, we
already have from __future__ import annotations everywhere. And getting
error like:
sqlalchemy.orm.exc.MappedAnnotationError: Could not resolve all types within
mapped annotation: "Mapped[int | None]"
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
mypy/ruff are not happy with Mapped[Optional[int]] (*ruff UP007*), and
SQLAlchemy 2.x cannot parse Mapped[int | None] on Python 3.9.
With from __future__ import annotations, SQLAlchemy stores the annotation
as a string. At runtime, it tries to eval() the string in the module's
namespace. On Python 3.9, eval("int | None") fails.
Need help and suggestion on how shall I proceed?
Thanks,
Avi
Re: [DISCUSS] Migrating to SQLAlchemy 2
Posted by Jens Scheffler <j_...@gmx.de.INVALID>.
FYI who was not on the dev call today.
We discussed about this but no majority was there to drop support for
Python 3.9 early. Rather the migration to SQLalchemy2 would be
post-poned to Airflow 3.2 or so as:
- The FAB dependency to 5.0 might need to be integrated first - and
SQLalchemy is not blocking this
- We would also need to draw support for 3.9 for providers and this has
a set of side-effects on monorepo and would be complex if we attempt to
keep 3.9 for providers only
- So rather no vote was the outcome.
If some other opinion please raise a [VOTE] thread.
On 08.05.25 08:59, Avi wrote:
> I would also favor dropping python 3.9 it would make a lot of code cleaner
> and easier to manage.
> Sure let's discuss in the dev call.
>
> - Avi
>
> On Thu, May 8, 2025 at 5:47 AM Amogh Desai <am...@gmail.com> wrote:
>
>> I would also be for bumping the minimum python version to 3.9+ for Airflow
>> 3.1 unless
>> there is some objection to this one.
>>
>> Maybe we should add this as a discussion item for the Airflow dev call?
>>
>> Thanks & Regards,
>> Amogh Desai
>>
>>
>> On Thu, May 8, 2025 at 11:09 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> I would be for it if that makes things easier. We could drop Python 3.9
>> for
>>> airflow 3.1+
>>>
>>> On Wed, May 7, 2025 at 10:01 PM Jens Scheffler
>> <j_scheffler@gmx.de.invalid
>>> wrote:
>>>
>>>> Hi Avi, all,
>>>>
>>>> we had a bit of a discussion about this and don't know if this is very
>>>> controversal... but: Python 3.9 will run out of support in September.
>>>> How about if we drop support early in 3.1 so that we migrate main to
>>>> 3.10 and by this remove the deadlock?
>>>>
>>>> Jens
>>>>
>>>> On 07.05.25 16:48, Abhishek Bhakat wrote:
>>>>> Hi All,
>>>>>
>>>>> I've hit a little deadlock.
>>>>>
>>>>> We want to use SQLAlchemy 2.x ORM typing (Mapped[...] =
>>>>> mapped_column(...)) with
>>>>> PEP 604 union types (int | None) in model annotations and must
>> support
>>>>> Python 3.9 (which does not support int | None at runtime).
>>>>>
>>>>> We do not want to use Optional[...] (ruff UP007 is enforced). And, we
>>>>> already have from __future__ import annotations everywhere. And
>> getting
>>>>> error like:
>>>>>
>>>>> sqlalchemy.orm.exc.MappedAnnotationError: Could not resolve all types
>>>> within
>>>>> mapped annotation: "Mapped[int | None]"
>>>>>
>>>>> TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
>>>>>
>>>>> mypy/ruff are not happy with Mapped[Optional[int]] (*ruff UP007*),
>> and
>>>>> SQLAlchemy 2.x cannot parse Mapped[int | None] on Python 3.9.
>>>>>
>>>>> With from __future__ import annotations, SQLAlchemy stores the
>>> annotation
>>>>> as a string. At runtime, it tries to eval() the string in the
>> module's
>>>>> namespace. On Python 3.9, eval("int | None") fails.
>>>>>
>>>>> Need help and suggestion on how shall I proceed?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Avi
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>>>> For additional commands, e-mail: dev-help@airflow.apache.org
>>>>
>>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org
Re: [DISCUSS] Migrating to SQLAlchemy 2
Posted by Avi <av...@astronomer.io.INVALID>.
I would also favor dropping python 3.9 it would make a lot of code cleaner
and easier to manage.
Sure let's discuss in the dev call.
- Avi
On Thu, May 8, 2025 at 5:47 AM Amogh Desai <am...@gmail.com> wrote:
> I would also be for bumping the minimum python version to 3.9+ for Airflow
> 3.1 unless
> there is some objection to this one.
>
> Maybe we should add this as a discussion item for the Airflow dev call?
>
> Thanks & Regards,
> Amogh Desai
>
>
> On Thu, May 8, 2025 at 11:09 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > I would be for it if that makes things easier. We could drop Python 3.9
> for
> > airflow 3.1+
> >
> > On Wed, May 7, 2025 at 10:01 PM Jens Scheffler
> <j_scheffler@gmx.de.invalid
> > >
> > wrote:
> >
> > > Hi Avi, all,
> > >
> > > we had a bit of a discussion about this and don't know if this is very
> > > controversal... but: Python 3.9 will run out of support in September.
> > > How about if we drop support early in 3.1 so that we migrate main to
> > > 3.10 and by this remove the deadlock?
> > >
> > > Jens
> > >
> > > On 07.05.25 16:48, Abhishek Bhakat wrote:
> > > > Hi All,
> > > >
> > > > I've hit a little deadlock.
> > > >
> > > > We want to use SQLAlchemy 2.x ORM typing (Mapped[...] =
> > > > mapped_column(...)) with
> > > > PEP 604 union types (int | None) in model annotations and must
> support
> > > > Python 3.9 (which does not support int | None at runtime).
> > > >
> > > > We do not want to use Optional[...] (ruff UP007 is enforced). And, we
> > > > already have from __future__ import annotations everywhere. And
> getting
> > > > error like:
> > > >
> > > > sqlalchemy.orm.exc.MappedAnnotationError: Could not resolve all types
> > > within
> > > > mapped annotation: "Mapped[int | None]"
> > > >
> > > > TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
> > > >
> > > > mypy/ruff are not happy with Mapped[Optional[int]] (*ruff UP007*),
> and
> > > > SQLAlchemy 2.x cannot parse Mapped[int | None] on Python 3.9.
> > > >
> > > > With from __future__ import annotations, SQLAlchemy stores the
> > annotation
> > > > as a string. At runtime, it tries to eval() the string in the
> module's
> > > > namespace. On Python 3.9, eval("int | None") fails.
> > > >
> > > > Need help and suggestion on how shall I proceed?
> > > >
> > > > Thanks,
> > > >
> > > > Avi
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>
Re: [DISCUSS] Migrating to SQLAlchemy 2
Posted by Amogh Desai <am...@gmail.com>.
I would also be for bumping the minimum python version to 3.9+ for Airflow
3.1 unless
there is some objection to this one.
Maybe we should add this as a discussion item for the Airflow dev call?
Thanks & Regards,
Amogh Desai
On Thu, May 8, 2025 at 11:09 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> I would be for it if that makes things easier. We could drop Python 3.9 for
> airflow 3.1+
>
> On Wed, May 7, 2025 at 10:01 PM Jens Scheffler <j_scheffler@gmx.de.invalid
> >
> wrote:
>
> > Hi Avi, all,
> >
> > we had a bit of a discussion about this and don't know if this is very
> > controversal... but: Python 3.9 will run out of support in September.
> > How about if we drop support early in 3.1 so that we migrate main to
> > 3.10 and by this remove the deadlock?
> >
> > Jens
> >
> > On 07.05.25 16:48, Abhishek Bhakat wrote:
> > > Hi All,
> > >
> > > I've hit a little deadlock.
> > >
> > > We want to use SQLAlchemy 2.x ORM typing (Mapped[...] =
> > > mapped_column(...)) with
> > > PEP 604 union types (int | None) in model annotations and must support
> > > Python 3.9 (which does not support int | None at runtime).
> > >
> > > We do not want to use Optional[...] (ruff UP007 is enforced). And, we
> > > already have from __future__ import annotations everywhere. And getting
> > > error like:
> > >
> > > sqlalchemy.orm.exc.MappedAnnotationError: Could not resolve all types
> > within
> > > mapped annotation: "Mapped[int | None]"
> > >
> > > TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
> > >
> > > mypy/ruff are not happy with Mapped[Optional[int]] (*ruff UP007*), and
> > > SQLAlchemy 2.x cannot parse Mapped[int | None] on Python 3.9.
> > >
> > > With from __future__ import annotations, SQLAlchemy stores the
> annotation
> > > as a string. At runtime, it tries to eval() the string in the module's
> > > namespace. On Python 3.9, eval("int | None") fails.
> > >
> > > Need help and suggestion on how shall I proceed?
> > >
> > > Thanks,
> > >
> > > Avi
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>
Re: [DISCUSS] Migrating to SQLAlchemy 2
Posted by Jarek Potiuk <ja...@potiuk.com>.
I would be for it if that makes things easier. We could drop Python 3.9 for
airflow 3.1+
On Wed, May 7, 2025 at 10:01 PM Jens Scheffler <j_...@gmx.de.invalid>
wrote:
> Hi Avi, all,
>
> we had a bit of a discussion about this and don't know if this is very
> controversal... but: Python 3.9 will run out of support in September.
> How about if we drop support early in 3.1 so that we migrate main to
> 3.10 and by this remove the deadlock?
>
> Jens
>
> On 07.05.25 16:48, Abhishek Bhakat wrote:
> > Hi All,
> >
> > I've hit a little deadlock.
> >
> > We want to use SQLAlchemy 2.x ORM typing (Mapped[...] =
> > mapped_column(...)) with
> > PEP 604 union types (int | None) in model annotations and must support
> > Python 3.9 (which does not support int | None at runtime).
> >
> > We do not want to use Optional[...] (ruff UP007 is enforced). And, we
> > already have from __future__ import annotations everywhere. And getting
> > error like:
> >
> > sqlalchemy.orm.exc.MappedAnnotationError: Could not resolve all types
> within
> > mapped annotation: "Mapped[int | None]"
> >
> > TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
> >
> > mypy/ruff are not happy with Mapped[Optional[int]] (*ruff UP007*), and
> > SQLAlchemy 2.x cannot parse Mapped[int | None] on Python 3.9.
> >
> > With from __future__ import annotations, SQLAlchemy stores the annotation
> > as a string. At runtime, it tries to eval() the string in the module's
> > namespace. On Python 3.9, eval("int | None") fails.
> >
> > Need help and suggestion on how shall I proceed?
> >
> > Thanks,
> >
> > Avi
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>
Re: [DISCUSS] Migrating to SQLAlchemy 2
Posted by Jens Scheffler <j_...@gmx.de.INVALID>.
Hi Avi, all,
we had a bit of a discussion about this and don't know if this is very
controversal... but: Python 3.9 will run out of support in September.
How about if we drop support early in 3.1 so that we migrate main to
3.10 and by this remove the deadlock?
Jens
On 07.05.25 16:48, Abhishek Bhakat wrote:
> Hi All,
>
> I've hit a little deadlock.
>
> We want to use SQLAlchemy 2.x ORM typing (Mapped[...] =
> mapped_column(...)) with
> PEP 604 union types (int | None) in model annotations and must support
> Python 3.9 (which does not support int | None at runtime).
>
> We do not want to use Optional[...] (ruff UP007 is enforced). And, we
> already have from __future__ import annotations everywhere. And getting
> error like:
>
> sqlalchemy.orm.exc.MappedAnnotationError: Could not resolve all types within
> mapped annotation: "Mapped[int | None]"
>
> TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
>
> mypy/ruff are not happy with Mapped[Optional[int]] (*ruff UP007*), and
> SQLAlchemy 2.x cannot parse Mapped[int | None] on Python 3.9.
>
> With from __future__ import annotations, SQLAlchemy stores the annotation
> as a string. At runtime, it tries to eval() the string in the module's
> namespace. On Python 3.9, eval("int | None") fails.
>
> Need help and suggestion on how shall I proceed?
>
> Thanks,
>
> Avi
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org