You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/08 09:24:34 UTC
[GitHub] [airflow] gastonlau opened a new issue #19462: ModuleNotFoundError when running airflow db init/upgrade
gastonlau opened a new issue #19462:
URL: https://github.com/apache/airflow/issues/19462
### Apache Airflow version
2.0.2
### Operating System
Amazon Linux
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon 1.4.0
apache-airflow-providers-ftp 2.0.1
apache-airflow-providers-http 1.1.1
apache-airflow-providers-imap 2.0.1
apache-airflow-providers-slack 3.0.0
apache-airflow-providers-sqlite 2.0.1
### Deployment
Virtualenv installation
### Deployment details
We are using the AWS CodeDeploy to deploy the DAG files and airflow.config as well as other necessary components to an AWS EC2 instance.
We are running the following steps every time when we run the deployment.
1. Install all Airflow/Python packages and dependencies using conda forge
2. Run `airflow db upgrade`
3. Start the webserver using `airflow webserver --daemon`
4. Start the scheduler using `airflow scheduler --daemon`
### What happened
I was deploying my DAGs to the EC2 instance as usual.
However, an error occurred today when running `airflow db upgrade`
```[2021-11-08 08:17:05,193] {db.py:684} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/airflow_env/bin/airflow", line 11, in <module>
sys.exit(main())
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 31, in initdb
db.initdb()
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/airflow/utils/db.py", line 559, in initdb
upgradedb()
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/airflow/utils/db.py", line 694, in upgradedb
command.upgrade(config, 'heads')
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/command.py", line 320, in upgrade
script.run_env()
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/script/base.py", line 563, in run_env
util.load_python_file(self.dir, "env.py")
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
module = load_module_py(module_id, path)
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/airflow/migrations/env.py", line 108, in <module>
run_migrations_online()
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/airflow/migrations/env.py", line 102, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
self.get_context().run_migrations(**kw)
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/runtime/migration.py", line 608, in run_migrations
for step in self._migrations_fn(heads, self):
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/command.py", line 309, in upgrade
return script._upgrade_revs(revision, rev)
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/script/base.py", line 439, in _upgrade_revs
for script in reversed(list(revs))
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/script/revision.py", line 793, in iterate_revisions
revisions, heads = fn(
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/script/revision.py", line 1393, in _collect_upgrade_revisions
targets: Collection["Revision"] = self._parse_upgrade_target(
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/script/revision.py", line 1193, in _parse_upgrade_target
return self.get_revisions(target)
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/script/revision.py", line 527, in get_revisions
resolved_id, branch_label = self._resolve_revision_number(
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/script/revision.py", line 747, in _resolve_revision_number
self._revision_map
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 893, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/script/revision.py", line 189, in _revision_map
for revision in self._generator():
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/script/base.py", line 136, in _load_revisions
script = Script._from_filename(self, vers, file_)
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/script/base.py", line 999, in _from_filename
module = util.load_python_file(dir_, filename)
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
module = load_module_py(module_id, path)
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/airflow/migrations/versions/2c6edca13270_resource_based_permissions.py", line 29, in <module>
from airflow.www.app import create_app
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/airflow/www/app.py", line 24, in <module>
from flask_appbuilder import SQLA
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/flask_appbuilder/__init__.py", line 6, in <module>
from .base import AppBuilder # noqa: F401
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/flask_appbuilder/base.py", line 8, in <module>
from .api.manager import OpenApiManager
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/flask_appbuilder/api/manager.py", line 7, in <module>
from flask_appbuilder.baseviews import BaseView
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/flask_appbuilder/baseviews.py", line 21, in <module>
from .forms import GeneralModelConverter
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/flask_appbuilder/forms.py", line 16, in <module>
from .fields import EnumField, QuerySelectField, QuerySelectMultipleField
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/flask_appbuilder/fields.py", line 6, in <module>
from wtforms.compat import string_types, text_type
ModuleNotFoundError: No module named 'wtforms.compat'
```
Running `airflow webserver --daemon` produces similar error
```
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/airflow_env/bin/airflow", line 11, in <module>
sys.exit(main())
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 47, in command
func = import_string(import_path)
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 32, in import_string
module = import_module(module_path)
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/airflow/cli/commands/webserver_command.py", line 43, in <module>
from airflow.www.app import cached_app, create_app
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/airflow/www/app.py", line 24, in <module>
from flask_appbuilder import SQLA
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/flask_appbuilder/__init__.py", line 6, in <module>
from .base import AppBuilder # noqa: F401
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/flask_appbuilder/base.py", line 8, in <module>
from .api.manager import OpenApiManager
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/flask_appbuilder/api/manager.py", line 7, in <module>
from flask_appbuilder.baseviews import BaseView
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/flask_appbuilder/baseviews.py", line 21, in <module>
from .forms import GeneralModelConverter
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/flask_appbuilder/forms.py", line 16, in <module>
from .fields import EnumField, QuerySelectField, QuerySelectMultipleField
File "/home/ec2-user/anaconda3/envs/airflow_env/lib/python3.8/site-packages/flask_appbuilder/fields.py", line 6, in <module>
from wtforms.compat import string_types, text_type
ModuleNotFoundError: No module named 'wtforms.compat'
```
Python version: 3.8.12
### What you expected to happen
I am expecting the airflow db migration should always be successful as I can see the related packages are already installed in pip3
WTForms==3.0.0
Flask-WTF==0.14.3
while I did not change any Airflow configuration except adding some DAG files.
The error have never happened before until today. The latest successful deployment is last Friday.
### How to reproduce
Running either one of these commands can reproduce this error `ModuleNotFoundError: No module named 'wtforms.compat'`
`airflow db init`
`airflow db upgrade`
`airflow db check-migrations`
`airflow webserver`
`airflow webserver --daemon`
### Anything else
I am trying not to test with the command `airflow db reset` since this will delete all the Airflow metadata.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #19462: ModuleNotFoundError when running airflow db init/upgrade
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #19462:
URL: https://github.com/apache/airflow/issues/19462#issuecomment-963003263
> WTForms==3.0.0
This version is not supported. Please use constraint files to install Airflow. http://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#constraints-files
For Airflow 2.0.2, you should use `WTForms==2.3.3`. See: https://github.com/apache/airflow/blob/constraints-2.0.2/constraints-3.8.txt#L35
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj closed issue #19462: ModuleNotFoundError when running airflow db init/upgrade
Posted by GitBox <gi...@apache.org>.
mik-laj closed issue #19462:
URL: https://github.com/apache/airflow/issues/19462
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org