You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2023/01/04 10:37:15 UTC
[GitHub] [airflow] potiuk opened a new issue, #28723: Fix all deprecations for SQLAlchemy 2.0
potiuk opened a new issue, #28723:
URL: https://github.com/apache/airflow/issues/28723
### Body
Airflow is currently not compatible with SQLAlchemy 2.0 which is about to be released. We need to make a deliberate effort
to support it.
Here are some info to aid in this effort:
* Description of all removed featuers in Airflow 2.0: https://sqlalche.me/e/b8d9
* The way to see all the deprecations in latest version of SQLAlchemy 1.4 is to set environment variable SQLALCHEMY_WARN_20=1
* How to start it:
- add SQLALCHEMY_WARN_20=1 to ci.yml at top level
- look through warnings.txt in tests to investigate the warnings (there will also be errors in the provider imports)
- when all of them are fixed - remove <2.0 limitation in sqlalchemy in setup.cfg
- make sure CI is green
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] phanikumv commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1652026556
Not sure of exact date - suggest to follow this [thread](https://apache-airflow.slack.com/archives/C03G9H97MM2/p1689624751944979)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1951454071
Why don you make a PR for that @Dev-iL ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1855694165
Just for the record, fix all stuff do not automatically granted ability to run Airflow with SQLAlchemy 2.0, due to upper bound limitation of Airflow and Providers dependencies:
```console
root@4c41f46e319c:/opt/airflow# pipdeptree --packages sqlalchemy -r
SQLAlchemy==1.4.50
├── alembic==1.13.0 [requires: SQLAlchemy>=1.3.0]
│ ├── apache-airflow==2.9.0.dev0 [requires: alembic>=1.6.3,<2.0]
│ ├── databricks-sql-connector==2.9.3 [requires: alembic>=1.0.11,<2.0.0]
│ └── sqlalchemy-spanner==1.6.2 [requires: alembic]
├── apache-airflow==2.9.0.dev0 [requires: SQLAlchemy>=1.4.28,<2.0]
├── databricks-sql-connector==2.9.3 [requires: SQLAlchemy>=1.3.24,<2.0.0]
├── eralchemy2==1.3.8 [requires: SQLAlchemy>=1.4]
├── Flask-AppBuilder==4.3.10 [requires: SQLAlchemy<1.5]
│ └── apache-airflow==2.9.0.dev0 [requires: Flask-AppBuilder==4.3.10]
├── Flask-SQLAlchemy==2.5.1 [requires: SQLAlchemy>=0.8.0]
│ └── Flask-AppBuilder==4.3.10 [requires: Flask-SQLAlchemy>=2.4,<3]
│ └── apache-airflow==2.9.0.dev0 [requires: Flask-AppBuilder==4.3.10]
├── marshmallow-sqlalchemy==0.26.1 [requires: SQLAlchemy>=1.2.0]
│ └── Flask-AppBuilder==4.3.10 [requires: marshmallow-sqlalchemy>=0.22.0,<0.27.0]
│ └── apache-airflow==2.9.0.dev0 [requires: Flask-AppBuilder==4.3.10]
├── snowflake-sqlalchemy==1.5.1 [requires: SQLAlchemy>=1.4.0,<2.0.0]
├── sqlalchemy-bigquery==1.9.0 [requires: SQLAlchemy>=1.2.0,<2.0.0dev]
├── sqlalchemy-drill==1.1.4 [requires: SQLAlchemy]
├── SQLAlchemy-JSONField==1.0.2 [requires: SQLAlchemy]
│ └── apache-airflow==2.9.0.dev0 [requires: SQLAlchemy-JSONField>=1.0]
├── sqlalchemy-redshift==0.8.14 [requires: SQLAlchemy>=0.9.2,<2.0.0]
├── sqlalchemy-spanner==1.6.2 [requires: SQLAlchemy>=1.1.13]
└── SQLAlchemy-Utils==0.41.1 [requires: SQLAlchemy>=1.3]
└── Flask-AppBuilder==4.3.10 [requires: SQLAlchemy-Utils>=0.32.21,<1]
└── apache-airflow==2.9.0.dev0 [requires: Flask-AppBuilder==4.3.10]
```
Also required that this packages also supports SA 2.0:
- [Flask-AppBuilder](https://pypi.org/project/Flask-AppBuilder/) - Doesn't support SA 2.0 yet
- [databricks-sql-connector](https://pypi.org/project/databricks-sql-connector/) - Support SA 2.0 from 3.0.0, and drop support 1.4
- [snowflake-sqlalchemy](https://pypi.org/project/snowflake-sqlalchemy/) - Doesn't support SA 2.0 yet
- [sqlalchemy-bigquery](https://pypi.org/project/sqlalchemy-bigquery/) - Doesn't support SA 2.0 yet
- [sqlalchemy-redshift](https://pypi.org/project/sqlalchemy-redshift/) - Doesn't support SA 2.0 yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-2052637964
Query API at that moment marked as legacy, and do not raise any warnings about future removal and there is become record about that this API will not removed from sqlalchemy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "Dev-iL (via GitHub)" <gi...@apache.org>.
Dev-iL commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1951460606
@potiuk Sure thing, will ping when ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "veinkr (via GitHub)" <gi...@apache.org>.
veinkr commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-2094746444
@Taragolis , sorry to bother, I jsut checked the [Flask-AppBuilder](https://pypi.org/project/Flask-AppBuilder/) repo, it haven't updated since two months ago, and it seems that it blocked by https://github.com/dpgaspar/Flask-AppBuilder/issues/2038#issuecomment-1549577483 so that it can't update to `flask-sqlalchemy >=3`.
Seems that we can't upgrade to sqlalchemy to 2.0 in a short time, do we have other plan for it?
I can see that in previous PR, the pandas have been limited to `pandas<2.2`, which is a important library for data engineer, if we keep sqlalchemy<2.0, then we will behind too much things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1385120239
Agree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1370941872
Just clarification. Our target support both 1.4.x and 2.0 or only 2.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-2094752721
The strategic, maybe even since Airflow 2.0, is get rid of FAB in Airflow at all.
It moved into the separate provider since 2.9, however all Authorisation still happen go through the [FAB](https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/index.html).
There is no ETA when it happen, more possible that in [Airflow 3.0](https://lists.apache.org/thread/3chvg9964zvh15mtrbl073f4oj3nlzp2) but maybe sooner if someone could propose how we could get rid off FAB "Here and Now"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1371038430
> Just clarification. Our target support both 1.4.x and 2.0 or only 2.0?
This is a good point, and I do not know the answer. I think it will entirely depend on:
a) when we will do it
b) how difficult it will be to keep compatibility
c) get some indication of the adoption of 2.0 once it is out there
I would refrain from trying to figure the answer before the final 2.0 release is out there - and possibly we should even wait for at least 2.0.1 or so, generally we should scout for stability indications. And then we should attempt to migrate and see how difficult it is to keep it compatible with both.
If we will decide to support both, due to many differences and generally backwards incompatible release of sqlalchemy I would be only heppy with supporting both if we extend our test suite in CI and add another test matrix dimention - sqlalchemy_version (on top of backend + backend version, Python version) that we have now.
But we will also have to see if that is really needed, depending on how complex the fixing will be and how different the two versions are. If the differences are small, we can likely get-by without having the extra matrix dimension.
If we see that differences are big, I think going 2.0-only is a better approach (and it basically means we will have to wait at least 6-10 months after release depending on adoption IMHO).
There is absolutely no hurry with migration yet, 1.4 will be out there for a long while and it will be supported with bugfixes - so we do not have (and will not have for a while) particular need to migrate to 2.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1407473316
> I think we should support both 1.4 and 2.0 for at least one minor release, preferrably much longer. Dependencies don’t tend to catch up very fast for this kind of migrations, and an Airflow installation generally has a lot of those.
Fully agreed, we need to support both
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1855569830
Hi Team!
Is there some progress on this one? Are we actually close to close it? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VladaZakharova commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1652018021
@phanikumv
thank you for the update :)
Do you know the approximate day when the new version will be released so we can continue work on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1371108400
I think we should support both 1.4 and 2.0 for at least one minor release, preferrably much longer. Dependencies don’t tend to catch up very fast for this kind of migrations, and an Airflow installation generally has a lot of those.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] infohash commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by "infohash (via GitHub)" <gi...@apache.org>.
infohash commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1456199003
What can be the expected version that will bring SQLAlchemy 2.0 support. I am making a service that will use SQLAlchemy 2.0 APIs. I will also integrate Airflow with that service. Based on ETA, I can decide whether to go ahead with SQLAlchemy 1.4 and not wait for the supported version of Airflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] avramdj commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by "avramdj (via GitHub)" <gi...@apache.org>.
avramdj commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1730120627
Hi everyone, what's the current state of this and is it known when we can expect SA2.0 support?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "moiseenkov (via GitHub)" <gi...@apache.org>.
moiseenkov commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1768132556
> Not yet @moiseenkov , we are close but still have couple of files which we are working on. I will keep you posted when all the codebase has been refactored !
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "infohash (via GitHub)" <gi...@apache.org>.
infohash commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1857972226
This is why I forked SQLAlchemy 2.0 and put it in my project's source directory instead of installing it as a project dependency. Once airflow ecosystem completely moves to SQLA 2.0, I will just stop using my own fork. All I had to do was change the package name of SQLA 2.0. You can also fork it and rebuild SQLA 2.0 wheel with a different package name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1768102141
Not yet @moiseenkov , we are close but still have couple of files which we are working on. I will keep you posted when all the codebase has been refactored !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "moiseenkov (via GitHub)" <gi...@apache.org>.
moiseenkov commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1768080376
Hi everyone, it seems that all deprecation warnings are suppressed already. Am I right?
If so, are we ready for the upgrade?
@Taragolis , @potiuk , @phanikumv
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] moiseenkov commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by "moiseenkov (via GitHub)" <gi...@apache.org>.
moiseenkov commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1649536712
Hi everyone,
Do we have some kind of board or issues list that are currently in progress? I see the work is going on, so I would like to help and fix some part that nobody else is working on at the moment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "dondaum (via GitHub)" <gi...@apache.org>.
dondaum commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-2067633235
Hi. I would like to support. To clarify is it correct that all warnings can be found in this issue detail section from potiuk?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "veinkr (via GitHub)" <gi...@apache.org>.
veinkr commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-2094701420
Hi, @dondaum , I saw you said that all `RemovedIn20Warning `should be fixed in PR [https://github.com/apache/airflow/pull/39299](https://github.com/apache/airflow/pull/39299) , does that mean we can upgrade the sqlalchemy to 2.0 version safely?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] moiseenkov commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by "moiseenkov (via GitHub)" <gi...@apache.org>.
moiseenkov commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1651518967
> Hi @VladaZakharova - we are currently suspending this effort till 2.7 is released. We have already completed the changes, but will keep the PR in draft mode until then. The other pending change is to fix the tests so that they use the sqlalchemy 2.0 style (but again they need to wait till 2.7 is released)
>
> cc @jedcunningham
@phanikumv ,
Thank you for replying. Could you please describe the reason for this suspension? Are there any blockers that will be resolved in 2.7?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-2052633482
Potentially if you able to run Airflow without FAB, the airflow potentially could starts with sqlalchemy 2. I’ve analyze Airflow warnings recently and unable to find any `sqlalchemy.exc.RemovedIn20Warnings` from the Airflow. Only FAB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-2051458814
Hi Team! How is the progress going in this one? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "Dev-iL (via GitHub)" <gi...@apache.org>.
Dev-iL commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1951118883
Not a deprecation but probably related to this effort: `DagBag` cannot be imported with SQLA 2:
```python
>>> from airflow.models import DagBag
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/.../.venv/lib64/python3.9/site-packages/airflow/__init__.py", line 68, in <module>
settings.initialize()
File "/home/.../.venv/lib64/python3.9/site-packages/airflow/settings.py", line 544, in initialize
configure_orm()
File "/home/.../.venv/lib64/python3.9/site-packages/airflow/settings.py", line 242, in configure_orm
engine = create_engine(SQL_ALCHEMY_CONN, connect_args=connect_args, **engine_args, future=True)
File "<string>", line 2, in create_engine
File "/home/.../.venv/lib64/python3.9/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
return fn(*args, **kwargs) # type: ignore[no-any-return]
File "/home/.../.venv/lib64/python3.9/site-packages/sqlalchemy/engine/create.py", line 686, in create_engine
raise TypeError(
TypeError: Invalid argument(s) 'encoding' sent to create_engine(), using configuration SQLiteDialect_pysqlite/QueuePool/Engine. Please check that the keyword arguments are appropriate for this combination of components.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-2066283481
I've add collect this warning during the CI step, however our warning collection system collect warnings only during run tests, and errors might happen during initial configurations, so this might be other incompatibilities, i would plan to extend our plugin and collect in the other steps but it required some time.
I have add all currently found warnings in Issue detail, so fill free to fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1457898605
It can be the next version if you help out on it 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] phanikumv commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1651448099
Hi @VladaZakharova - we are currently suspending this effort till 2.7 is released. We have already completed the changes, but will keep the PR in draft mode until then. The other pending change is to fix the tests so that they use the sqlalchemy 2.0 style (but again they need to wait till 2.7 is released)
cc @jedcunningham
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vumdao commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by "vumdao (via GitHub)" <gi...@apache.org>.
vumdao commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1656818098
It's better to throw a warning on https://airflow.apache.org/docs/apache-airflow/stable/security/flower.html#flower so that people aware the issue when trying to add Flower component.
```
/home/airflow/.local/lib/python3.7/site-packages/airflow/models/base.py:49 MovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings. Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
ERROR: You need to upgrade the database. Please run `airflow db upgrade`. Make sure the command is run using Airflow version 2.5.3.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] phanikumv commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by "phanikumv (via GitHub)" <gi...@apache.org>.
phanikumv commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1652004364
Hey @moiseenkov since we are prepping for 2.7 release, we want to ensure that the main branch doesn't have anymore merges which might impact core airflow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1371092680
Migrate 2.0 to 1.4 potentially could have more effect rather than we have with other packages.
Some of dependency of providers also use SQLAlchemy and we need to wait until them migrate to 2.0
Current packages which explicit have SQLAlchemy as dependency from CI image.
```console
root@6e6ad04bdb9b:/opt/airflow# pipdeptree --reverse --packages SQLAlchemy --python `which python`
sqlalchemy==1.4.46
- alembic==1.9.1 [requires: SQLAlchemy>=1.3.0]
- apache-airflow==2.6.0.dev0 [requires: alembic>=1.6.3,<2.0]
- apache-airflow==2.6.0.dev0 [requires: sqlalchemy>=1.4,<2.0]
- elasticsearch-dbapi==0.2.9 [requires: sqlalchemy]
- eralchemy2==1.3.6 [requires: SQLAlchemy>=1.3.19]
- Flask-AppBuilder==4.1.4 [requires: SQLAlchemy<1.5]
- apache-airflow==2.6.0.dev0 [requires: flask-appbuilder==4.1.4]
- Flask-SQLAlchemy==2.5.1 [requires: SQLAlchemy>=0.8.0]
- Flask-AppBuilder==4.1.4 [requires: Flask-SQLAlchemy>=2.4,<3]
- apache-airflow==2.6.0.dev0 [requires: flask-appbuilder==4.1.4]
- marshmallow-sqlalchemy==0.26.1 [requires: SQLAlchemy>=1.2.0]
- Flask-AppBuilder==4.1.4 [requires: marshmallow-sqlalchemy>=0.22.0,<0.27.0]
- apache-airflow==2.6.0.dev0 [requires: flask-appbuilder==4.1.4]
- snowflake-sqlalchemy==1.4.4 [requires: sqlalchemy>=1.4.0,<2.0.0]
- sqlalchemy-bigquery==1.5.0 [requires: sqlalchemy>=1.2.0,<2.0.0dev]
- sqlalchemy-drill==1.1.2 [requires: sqlalchemy]
- SQLAlchemy-JSONField==1.0.1.post0 [requires: sqlalchemy]
- apache-airflow==2.6.0.dev0 [requires: sqlalchemy-jsonfield>=1.0]
- sqlalchemy-redshift==0.8.12 [requires: SQLAlchemy>=0.9.2,<2.0.0]
- SQLAlchemy-Utils==0.39.0 [requires: SQLAlchemy>=1.3]
- Flask-AppBuilder==4.1.4 [requires: sqlalchemy-utils>=0.32.21,<1]
- apache-airflow==2.6.0.dev0 [requires: flask-appbuilder==4.1.4]
```
This is not complete list because some of packages could use sqlalchemy but do not have it in dependency, for example `pandas` use it in [read_sql](https://pandas.pydata.org/docs/reference/api/pandas.read_sql.html#pandas-read-sql).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1855600387
I don't think that someone track progress for this one due to huge scope of the task/issue. However as I could see a lot of stuff migrated from the Query API to the statement based builder (or how it correctly named).
There is still remaining parts exists in (this **might not** a complete list):
- In FAB provider: `airflow/providers/fab/`
- Migration scripts: `airflow/migrations/versions/`
- In selected modules:
- `airflow/dag_processing/processor.py`
- `airflow/models/taskinstance.py`
- `airflow/sensors/external_task.py`
- `airflow/triggers/external_task.py`
- `airflow/utils/db_cleanup.py`
- `airflow/utils/log/file_task_handler.py`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VladaZakharova commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1651236847
@kaxil @phanikumv @uranusjr @hussein-awala
Maybe you could help with coordination here? Do you have any plan for the process here?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VladaZakharova commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1652056317
Got it! Lets then be in touch about the updates here to continue work on this issue :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] auvipy commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by "auvipy (via GitHub)" <gi...@apache.org>.
auvipy commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1464864672
I want to contribute here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] auvipy commented on issue #28723: Fix all deprecations for SQLAlchemy 2.0
Posted by "auvipy (via GitHub)" <gi...@apache.org>.
auvipy commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-1464867113
SQLA 1.4 & 2.0 both can be supported at the same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "moiseenkov (via GitHub)" <gi...@apache.org>.
moiseenkov commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-2058451579
Excluding providers (FAB and openlineage) I managed to catch two warnings in the core:
```
airflow/www/views.py:845: [W0513(warning), ] The "columns" argument to Select.with_only_columns(), when referring to a sequence of items, is now passed as a series of positional elements, rather than as a list. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
airflow/models/taskinstance.py:1870: [W0513(warning), ] Using strings to indicate column or relationship paths in loader options is deprecated and will be removed in SQLAlchemy 2.0. Please use the class-bound attribute directly. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Fix all deprecations for SQLAlchemy 2.0 [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #28723:
URL: https://github.com/apache/airflow/issues/28723#issuecomment-2094724901
Not really, see this comment https://github.com/apache/airflow/issues/28723#issuecomment-1855694165
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org