You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Tenzer (via GitHub)" <gi...@apache.org> on 2024/01/19 16:19:08 UTC
[I] Airflow 2.8.1 contains migration requiring SQLAlchemy 2+ [airflow]
Tenzer opened a new issue, #36912:
URL: https://github.com/apache/airflow/issues/36912
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.8.1
### What happened?
I've upgraded our Airflow installation to the newly released version 2.8.1 and ran `airflow db upgrade`, as prompted by the message when Airflow tries to start up.
That, however, resulted in the following stack trace:
```
[2024-01-19T16:06:41.883+0000] {migration.py:213} INFO - Context impl MySQLImpl.
[2024-01-19T16:06:41.886+0000] {migration.py:216} INFO - Will assume non-transactional DDL.
[2024-01-19T16:06:41.897+0000] {db.py:1616} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade 10b52ebd31f7 -> 88344c1d9134, Drop unused TI index
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.11/site-packages/airflow/__main__.py", line 57, in main
args.func(args)
File "/usr/local/lib/python3.11/site-packages/airflow/cli/cli_config.py", line 49, in command
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/airflow/cli/commands/db_command.py", line 69, in upgradedb
migratedb(args)
File "/usr/local/lib/python3.11/site-packages/airflow/utils/cli.py", line 114, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/airflow/cli/commands/db_command.py", line 129, in migratedb
db.upgradedb(
File "/usr/local/lib/python3.11/site-packages/airflow/utils/session.py", line 79, in wrapper
return func(*args, session=session, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/airflow/utils/db.py", line 1623, in upgradedb
command.upgrade(config, revision=to_revision or "heads")
File "/usr/local/lib/python3.11/site-packages/alembic/command.py", line 399, in upgrade
script.run_env()
File "/usr/local/lib/python3.11/site-packages/alembic/script/base.py", line 578, in run_env
util.load_python_file(self.dir, "env.py")
File "/usr/local/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
module = load_module_py(module_id, path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/alembic/util/pyfiles.py", line 109, in load_module_py
spec.loader.exec_module(module) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/usr/local/lib/python3.11/site-packages/airflow/migrations/env.py", line 117, in <module>
run_migrations_online()
File "/usr/local/lib/python3.11/site-packages/airflow/migrations/env.py", line 111, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/usr/local/lib/python3.11/site-packages/alembic/runtime/environment.py", line 937, in run_migrations
self.get_context().run_migrations(**kw)
File "/usr/local/lib/python3.11/site-packages/alembic/runtime/migration.py", line 624, in run_migrations
step.migration_fn(**kw)
File "/usr/local/lib/python3.11/site-packages/airflow/migrations/versions/0133_2_8_1_refactor_dag_run_indexes.py", line 46, in upgrade
op.drop_index("ti_state_incl_start_date", table_name="task_instance", if_exists=True)
File "<string>", line 8, in drop_index
File "<string>", line 3, in drop_index
File "/usr/local/lib/python3.11/site-packages/alembic/operations/ops.py", line 1122, in drop_index
return operations.invoke(op)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/alembic/operations/base.py", line 393, in invoke
return fn(self, operation)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/alembic/operations/toimpl.py", line 113, in drop_index
raise NotImplementedError("SQLAlchemy 2.0+ required")
NotImplementedError: SQLAlchemy 2.0+ required
```,
### What you think should happen instead?
Ideally, the migration should run without issues, using a version of SQLAlchemy that Airflow supports.
### How to reproduce
Upgrade an existing Airflow installation to version 2.8.1. Run `airflow db upgrade`.
### Operating System
Docker
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.16.0
apache-airflow-providers-celery==3.4.1
apache-airflow-providers-common-io==1.2.0
apache-airflow-providers-common-sql==1.7.2
apache-airflow-providers-ftp==3.5.2
apache-airflow-providers-google==10.13.1
apache-airflow-providers-http==4.5.2
apache-airflow-providers-imap==3.3.2
apache-airflow-providers-mysql==5.5.1
apache-airflow-providers-sqlite==3.4.3
### Deployment
Other Docker-based deployment
### Deployment details
_No response_
### Anything else?
SQLAlchemy related packages:
Flask-SQLAlchemy==2.5.1
marshmallow-sqlalchemy==0.26.1
SQLAlchemy==1.4.49
sqlalchemy-bigquery==1.8.0
SQLAlchemy-JSONField==1.0.1.post0
sqlalchemy-redshift==0.8.14
sqlalchemy-spanner==1.6.2
SQLAlchemy-Utils==0.41.1
Alembic is at version 1.12.0.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 2.8.1 contains migration requiring SQLAlchemy 2+ [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #36912:
URL: https://github.com/apache/airflow/issues/36912#issuecomment-1900709515
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 2.8.1 contains migration requiring SQLAlchemy 2+ [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #36912:
URL: https://github.com/apache/airflow/issues/36912#issuecomment-1902037155
> What a bizarre setup. I guess I'll have to see if I can somehow get that shoehorned into how we manage dependencies. Thank you for the pointer,.
It's not bizarre, it's just. It is just designed for both - allowing for reproducible installation AND allowing people to upgrede depedencies when they neeed. In a few years a form of it will become a norm, simply the Python packaging and installation world did not yet feel then need for it, but with 670 dependencies of Airflow, this saves our users a lot of headaches, while allowing them also to use newer version of libraries. If they follow our advices that is.
If you want to know more about and have more context - I recommend to watch my talk from Airlfow Summit https://airflowsummit.org/sessions/2023/mastering-dependencies-the-airflow-way/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 2.8.1 contains migration requiring SQLAlchemy 2+ [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #36912:
URL: https://github.com/apache/airflow/issues/36912#issuecomment-1902038189
BTW. yes. I agree, likely adding alembic min version in this case is a good idea as well. Alembic has very little dependencies on its own so this error is a good reason to add min-version for 2.8.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 2.8.1 contains migration requiring SQLAlchemy 2+ [airflow]
Posted by "Tenzer (via GitHub)" <gi...@apache.org>.
Tenzer commented on issue #36912:
URL: https://github.com/apache/airflow/issues/36912#issuecomment-1900734293
I took a look at the [Alembic changelog](https://alembic.sqlalchemy.org/en/latest/changelog.html) and realised 1.13.0 fixes a bug where some operations by mistake were marked as requiring SQLAlchemy 2.0+.
Upgrading our version of Alembic from 1.12.0 to the latest 1.13.1 version also fixes this problem.
Would it perhaps be a good idea to require the Alembic version to be at minimum version 1.13.0 so other people don't bump into this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 2.8.1 contains migration requiring SQLAlchemy 2+ [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis closed issue #36912: Airflow 2.8.1 contains migration requiring SQLAlchemy 2+
URL: https://github.com/apache/airflow/issues/36912
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 2.8.1 contains migration requiring SQLAlchemy 2+ [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #36912:
URL: https://github.com/apache/airflow/issues/36912#issuecomment-1900772724
You need to install Airflow by provide constraints for specific Airflow and Python versions, for example in your case
```console
pip install "apache-airflow==2.8.1” --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.11.txt"
```
This action will prevent surprises when upstream package is (not)updated and breaks some critical part of the Airflow.
https://github.com/apache/airflow/blob/ae9614db29798b45b47da69f30f4f92ce8f0c8ae/constraints-3.11.txt#L77-L79
For the reference:
- [Installation from PyPI](https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html)
- [[Reminder] How to reproducibly install Airflow](https://lists.apache.org/thread/b8q1t3pvtky7y7mb2kjt4r2bz5bvz8qy)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Airflow 2.8.1 contains migration requiring SQLAlchemy 2+ [airflow]
Posted by "Tenzer (via GitHub)" <gi...@apache.org>.
Tenzer commented on issue #36912:
URL: https://github.com/apache/airflow/issues/36912#issuecomment-1900780574
What a bizarre setup. I guess I'll have to see if I can somehow get that shoehorned into how we manage dependencies. Thank you for the pointer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org