You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "eselyavka (via GitHub)" <gi...@apache.org> on 2023/11/02 17:57:39 UTC
[I] airflow db migrate crashing with new connexion == 3.0.0 [airflow]
eselyavka opened a new issue, #35375:
URL: https://github.com/apache/airflow/issues/35375
### Apache Airflow version
2.7.2
### What happened
Team, it looks like i just discovered bug in airflow 2.7.2(latest) which were introduced by this package https://pypi.org/project/connexion/3.0.0/, which were released several hours ago
I am running my airflow image in ubuntu 20.04 x86_64 docker image under the python 3.9.16
```
root@cbe8e0b21de3:/# python3.9 --version
Python 3.9.18
```
I am installing airflow via pip
```
pip install apache-airflow=2.7.2
```
I am seeing this error when performing `airflow db migrate`
```
root@cbe8e0b21de3:/# airflow db init
/usr/local/lib/python3.9/dist-packages/airflow/cli/commands/db_command.py:43 DeprecationWarning: `db init` is deprecated. Use `db migrate` instead to migrate the db and/or airflow connections create-default-connections to create the default connections
DB: sqlite:////root/airflow/airflow.db
[2023-11-02T13:54:50.283-0400] {migration.py:213} INFO - Context impl SQLiteImpl.
[2023-11-02T13:54:50.285-0400] {migration.py:216} INFO - Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.9/dist-packages/airflow/__main__.py", line 59, in main
args.func(args)
File "/usr/local/lib/python3.9/dist-packages/airflow/cli/cli_config.py", line 49, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
return func(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/airflow/cli/commands/db_command.py", line 49, in initdb
db.initdb()
File "/usr/local/lib/python3.9/dist-packages/airflow/utils/session.py", line 77, in wrapper
return func(*args, session=session, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/airflow/utils/db.py", line 739, in initdb
_create_db_from_orm(session=session)
File "/usr/local/lib/python3.9/dist-packages/airflow/utils/db.py", line 727, in _create_db_from_orm
command.stamp(config, "head")
File "/usr/local/lib/python3.9/dist-packages/alembic/command.py", line 685, in stamp
script.run_env()
File "/usr/local/lib/python3.9/dist-packages/alembic/script/base.py", line 579, in run_env
util.load_python_file(self.dir, "env.py")
File "/usr/local/lib/python3.9/dist-packages/alembic/util/pyfiles.py", line 93, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.9/dist-packages/alembic/util/pyfiles.py", line 109, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/usr/local/lib/python3.9/dist-packages/airflow/migrations/env.py", line 117, in <module>
run_migrations_online()
File "/usr/local/lib/python3.9/dist-packages/airflow/migrations/env.py", line 111, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/usr/local/lib/python3.9/dist-packages/alembic/runtime/environment.py", line 938, in run_migrations
self.get_context().run_migrations(**kw)
File "/usr/local/lib/python3.9/dist-packages/alembic/runtime/migration.py", line 612, in run_migrations
for step in self._migrations_fn(heads, self):
File "/usr/local/lib/python3.9/dist-packages/alembic/command.py", line 673, in do_stamp
return script._stamp_revs(util.to_tuple(destination_revs), rev)
File "/usr/local/lib/python3.9/dist-packages/alembic/script/base.py", line 494, in _stamp_revs
self.revision_map.filter_for_lineage(
File "/usr/local/lib/python3.9/dist-packages/alembic/script/revision.py", line 680, in filter_for_lineage
id_, branch_label = self._resolve_revision_number(check_against)
File "/usr/local/lib/python3.9/dist-packages/alembic/script/revision.py", line 754, in _resolve_revision_number
self._revision_map
File "/usr/local/lib/python3.9/dist-packages/sqlalchemy/util/langhelpers.py", line 1113, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "/usr/local/lib/python3.9/dist-packages/alembic/script/revision.py", line 197, in _revision_map
for revision in self._generator():
File "/usr/local/lib/python3.9/dist-packages/alembic/script/base.py", line 149, in _load_revisions
script = Script._from_filename(self, dir_name, filename)
File "/usr/local/lib/python3.9/dist-packages/alembic/script/base.py", line 1035, in _from_filename
module = util.load_python_file(dir_, filename)
File "/usr/local/lib/python3.9/dist-packages/alembic/util/pyfiles.py", line 93, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.9/dist-packages/alembic/util/pyfiles.py", line 109, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/usr/local/lib/python3.9/dist-packages/airflow/migrations/versions/0074_2_0_0_resource_based_permissions.py", line 30, in <module>
from airflow.www.app import cached_app
File "/usr/local/lib/python3.9/dist-packages/airflow/www/app.py", line 50, in <module>
from airflow.www.extensions.init_views import (
File "/usr/local/lib/python3.9/dist-packages/airflow/www/extensions/init_views.py", line 27, in <module>
from connexion.decorators.validation import RequestBodyValidator
ModuleNotFoundError: No module named 'connexion.decorators.validation'
```
If i downgrade connexion to 2.14.2 db migration command succeed
```
root@cbe8e0b21de3:/# pip3.9 install connexion==2.14.2
Collecting connexion==2.14.2
Downloading connexion-2.14.2-py2.py3-none-any.whl (95 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.1/95.1 kB 3.5 MB/s eta 0:00:00
Collecting clickclick<21,>=1.2 (from connexion==2.14.2)
Downloading clickclick-20.10.2-py2.py3-none-any.whl (7.4 kB)
Requirement already satisfied: jsonschema<5,>=2.5.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (4.19.2)
Requirement already satisfied: PyYAML<7,>=5.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (6.0.1)
Requirement already satisfied: requests<3,>=2.9.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.31.0)
Requirement already satisfied: inflection<0.6,>=0.3.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (0.5.1)
Collecting werkzeug<2.3,>=1.0 (from connexion==2.14.2)
Downloading Werkzeug-2.2.3-py3-none-any.whl (233 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 233.6/233.6 kB 8.1 MB/s eta 0:00:00
Requirement already satisfied: packaging>=20 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (23.2)
Requirement already satisfied: flask<2.3,>=1.0.4 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.2.5)
Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.1.2)
Requirement already satisfied: click>=4.0 in /usr/local/lib/python3.9/dist-packages (from clickclick<21,>=1.2->connexion==2.14.2) (8.1.7)
Requirement already satisfied: Jinja2>=3.0 in /usr/local/lib/python3.9/dist-packages (from flask<2.3,>=1.0.4->connexion==2.14.2) (3.1.2)
Requirement already satisfied: importlib-metadata>=3.6.0 in /usr/local/lib/python3.9/dist-packages (from flask<2.3,>=1.0.4->connexion==2.14.2) (6.8.0)
Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (23.1.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (2023.7.1)
Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (0.30.2)
Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (0.10.6)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.9/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (2.8)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (1.25.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (2019.11.28)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.9/dist-packages (from werkzeug<2.3,>=1.0->connexion==2.14.2) (2.1.3)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.9/dist-packages (from importlib-metadata>=3.6.0->flask<2.3,>=1.0.4->connexion==2.14.2) (3.17.0)
Installing collected packages: werkzeug, clickclick, connexion
Attempting uninstall: werkzeug
Found existing installation: Werkzeug 3.0.1
Uninstalling Werkzeug-3.0.1:
Successfully uninstalled Werkzeug-3.0.1
Attempting uninstall: connexion
Found existing installation: connexion 3.0.0
Uninstalling connexion-3.0.0:
Successfully uninstalled connexion-3.0.0
Successfully installed clickclick-20.10.2 connexion-2.14.2 werkzeug-2.2.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@cbe8e0b21de3:/# airflow db init
/usr/local/lib/python3.9/dist-packages/airflow/cli/commands/db_command.py:43 DeprecationWarning: `db init` is deprecated. Use `db migrate` instead to migrate the db and/or airflow connections create-default-connections to create the default connections
DB: sqlite:////root/airflow/airflow.db
[2023-11-02T13:56:12.691-0400] {migration.py:213} INFO - Context impl SQLiteImpl.
[2023-11-02T13:56:12.714-0400] {migration.py:216} INFO - Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running stamp_revision -> 405de8318b3a
WARNI [airflow.models.crypto] empty cryptography key - values will not be stored encrypted.
Initialization done
root@cbe8e0b21de3:/#
```
### What you think should happen instead
```
root@cbe8e0b21de3:/# pip3.9 install connexion==2.14.2
Collecting connexion==2.14.2
Downloading connexion-2.14.2-py2.py3-none-any.whl (95 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.1/95.1 kB 3.5 MB/s eta 0:00:00
Collecting clickclick<21,>=1.2 (from connexion==2.14.2)
Downloading clickclick-20.10.2-py2.py3-none-any.whl (7.4 kB)
Requirement already satisfied: jsonschema<5,>=2.5.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (4.19.2)
Requirement already satisfied: PyYAML<7,>=5.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (6.0.1)
Requirement already satisfied: requests<3,>=2.9.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.31.0)
Requirement already satisfied: inflection<0.6,>=0.3.1 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (0.5.1)
Collecting werkzeug<2.3,>=1.0 (from connexion==2.14.2)
Downloading Werkzeug-2.2.3-py3-none-any.whl (233 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 233.6/233.6 kB 8.1 MB/s eta 0:00:00
Requirement already satisfied: packaging>=20 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (23.2)
Requirement already satisfied: flask<2.3,>=1.0.4 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.2.5)
Requirement already satisfied: itsdangerous>=0.24 in /usr/local/lib/python3.9/dist-packages (from connexion==2.14.2) (2.1.2)
Requirement already satisfied: click>=4.0 in /usr/local/lib/python3.9/dist-packages (from clickclick<21,>=1.2->connexion==2.14.2) (8.1.7)
Requirement already satisfied: Jinja2>=3.0 in /usr/local/lib/python3.9/dist-packages (from flask<2.3,>=1.0.4->connexion==2.14.2) (3.1.2)
Requirement already satisfied: importlib-metadata>=3.6.0 in /usr/local/lib/python3.9/dist-packages (from flask<2.3,>=1.0.4->connexion==2.14.2) (6.8.0)
Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (23.1.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (2023.7.1)
Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (0.30.2)
Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5,>=2.5.1->connexion==2.14.2) (0.10.6)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.9/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (2.8)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (1.25.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests<3,>=2.9.1->connexion==2.14.2) (2019.11.28)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.9/dist-packages (from werkzeug<2.3,>=1.0->connexion==2.14.2) (2.1.3)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.9/dist-packages (from importlib-metadata>=3.6.0->flask<2.3,>=1.0.4->connexion==2.14.2) (3.17.0)
Installing collected packages: werkzeug, clickclick, connexion
Attempting uninstall: werkzeug
Found existing installation: Werkzeug 3.0.1
Uninstalling Werkzeug-3.0.1:
Successfully uninstalled Werkzeug-3.0.1
Attempting uninstall: connexion
Found existing installation: connexion 3.0.0
Uninstalling connexion-3.0.0:
Successfully uninstalled connexion-3.0.0
Successfully installed clickclick-20.10.2 connexion-2.14.2 werkzeug-2.2.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@cbe8e0b21de3:/# airflow db init
/usr/local/lib/python3.9/dist-packages/airflow/cli/commands/db_command.py:43 DeprecationWarning: `db init` is deprecated. Use `db migrate` instead to migrate the db and/or airflow connections create-default-connections to create the default connections
DB: sqlite:////root/airflow/airflow.db
[2023-11-02T13:56:12.691-0400] {migration.py:213} INFO - Context impl SQLiteImpl.
[2023-11-02T13:56:12.714-0400] {migration.py:216} INFO - Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running stamp_revision -> 405de8318b3a
WARNI [airflow.models.crypto] empty cryptography key - values will not be stored encrypted.
Initialization done
root@cbe8e0b21de3:/#
```
### How to reproduce
docker run -it --rm --entrypoint /bin/bash ubuntu:focal
apt install software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt install python3.9 gcc python3.9-dev
apt install curl
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
apt install python3.9-distutils
python3.9 get-pip.py
pip install apache-airflow==2.7.2
airflow db init
### Operating System
NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal
### Versions of Apache Airflow Providers
pip3.9 install apache-airflow==2.7.2
### Deployment
Other
### Deployment details
docker run -it --rm --entrypoint /bin/bash ubuntu:focal
apt update
apt install software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt update
apt install curl python3.9 gcc python3.9-dev python3.9-distutils
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3.9 get-pip.py
pip install apache-airflow==2.7.2
airflow db init
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow db migrate crashing with new connexion == 3.0.0 [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis closed issue #35375: airflow db migrate crashing with new connexion == 3.0.0
URL: https://github.com/apache/airflow/issues/35375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow db migrate crashing with new connexion == 3.0.0 [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #35375:
URL: https://github.com/apache/airflow/issues/35375#issuecomment-1791267540
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow db migrate crashing with new connexion == 3.0.0 [airflow]
Posted by "eselyavka (via GitHub)" <gi...@apache.org>.
eselyavka commented on issue #35375:
URL: https://github.com/apache/airflow/issues/35375#issuecomment-1808487316
@potiuk thank you very much for detailed answer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow db migrate crashing with new connexion == 3.0.0 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #35375:
URL: https://github.com/apache/airflow/issues/35375#issuecomment-1791529020
Also @eselyavka you should install airflow from the scratch (if you want reproducible install) following the only supported method - i.e. with constraints - https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html
Then you can upgrade any new dependencies as you see fit (and test) but the "base" installation is guaranteed to not cause problems when external dependencies are changing something in their behaviours
It is SPECIFICALLY designed to make sure that such breaking releases in "future" dependencies are not affecting user installations - for users who actually follow those instructions that is. You wil make yourself a favour to use the protections we implemented for our users.
Also you might want to watch my talk https://www.youtube.com/watch?v=zPjIQjjjyHI describing why and how it works if you are interested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow db migrate crashing with new connexion == 3.0.0 [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #35375:
URL: https://github.com/apache/airflow/issues/35375#issuecomment-1791285661
connexion already limited in:
- https://github.com/apache/airflow/pull/35218
Follow up task exists:
- https://github.com/apache/airflow/issues/35234
For avoid further error on any new package updates which might potentially break airflow it is recommend way to use [constraints file](https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#constraints-files)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org