You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/20 05:26:21 UTC
[GitHub] [airflow] zhzhang opened a new issue #15447: `db init` and `webserver` commands do not work on fresh install without constraints
zhzhang opened a new issue #15447:
URL: https://github.com/apache/airflow/issues/15447
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**: 2.0.1
**Environment**:
- **OS** (e.g. from /etc/os-release): OSX Catalina 10.15.7
- **Install tools**: Either poetry or pip
- **Others**: Python 3.7.10
**What happened**:
I make a fresh install of Airflow 2.0.1 in a new venv with `python3 -m pip install apache-airflow==2.0.1` (or alternatively `poetry add apache-airflow==2.0.1` in a new poetry environment).
I then try to run `airflow db init` and receive the following error:
```
[2021-04-19 21:46:04,373] {abstract.py:229} ERROR - Failed to add operation for GET /api/v1/connections
Traceback (most recent call last):
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/connexion/apis/abstract.py", line 209, in add_paths
self.add_operation(path, method)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/connexion/apis/abstract.py", line 173, in add_operation
pass_context_arg_name=self.pass_context_arg_name
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/connexion/operations/__init__.py", line 8, in make_operation
return spec.operation_cls.from_spec(spec, *args, **kwargs)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/connexion/operations/openapi.py", line 138, in from_spec
**kwargs
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/connexion/operations/openapi.py", line 89, in __init__
pass_context_arg_name=pass_context_arg_name
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/connexion/operations/abstract.py", line 96, in __init__
self._resolution = resolver.resolve(self)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/connexion/resolver.py", line 40, in resolve
return Resolution(self.resolve_function_from_operation_id(operation_id), operation_id)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/connexion/resolver.py", line 66, in resolve_function_from_operation_id
raise ResolverError(str(e), sys.exc_info())
connexion.exceptions.ResolverError: <ResolverError: columns>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/jorzhang/tmp2/venv/bin/airflow", line 8, in <module>
sys.exit(main())
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/airflow/cli/commands/db_command.py", line 31, in initdb
db.initdb()
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/airflow/utils/db.py", line 549, in initdb
upgradedb()
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/airflow/utils/db.py", line 684, in upgradedb
command.upgrade(config, 'heads')
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/alembic/command.py", line 294, in upgrade
script.run_env()
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/alembic/script/base.py", line 490, in run_env
util.load_python_file(self.dir, "env.py")
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
module = load_module_py(module_id, path)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/alembic/util/compat.py", line 182, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/airflow/migrations/env.py", line 108, in <module>
run_migrations_online()
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/airflow/migrations/env.py", line 102, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/alembic/runtime/environment.py", line 813, in run_migrations
self.get_context().run_migrations(**kw)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/alembic/runtime/migration.py", line 560, in run_migrations
step.migration_fn(**kw)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/airflow/migrations/versions/2c6edca13270_resource_based_permissions.py", line 314, in upgrade
remap_permissions()
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/airflow/migrations/versions/2c6edca13270_resource_based_permissions.py", line 289, in remap_permissions
appbuilder = create_app(config={'FAB_UPDATE_PERMS': False}).appbuilder
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/airflow/www/app.py", line 120, in create_app
init_api_connexion(flask_app)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/airflow/www/extensions/init_views.py", line 172, in init_api_connexion
specification='v1.yaml', base_path=base_path, validate_responses=True, strict_validation=True
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/connexion/apps/flask_app.py", line 57, in add_api
api = super(FlaskApp, self).add_api(specification, **kwargs)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/connexion/apps/abstract.py", line 156, in add_api
options=api_options.as_dict())
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/connexion/apis/abstract.py", line 111, in __init__
self.add_paths()
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/connexion/apis/abstract.py", line 216, in add_paths
self._handle_add_operation_error(path, method, err.exc_info)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/connexion/apis/abstract.py", line 231, in _handle_add_operation_error
raise value.with_traceback(traceback)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/connexion/resolver.py", line 61, in resolve_function_from_operation_id
return self.function_resolver(operation_id)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/connexion/utils.py", line 111, in get_function_from_name
module = importlib.import_module(module_name)
File "/Users/jorzhang/.pyenv/versions/3.7.10/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/airflow/api_connexion/endpoints/connection_endpoint.py", line 26, in <module>
from airflow.api_connexion.schemas.connection_schema import (
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/airflow/api_connexion/schemas/connection_schema.py", line 42, in <module>
class ConnectionSchema(ConnectionCollectionItemSchema): # pylint: disable=too-many-ancestors
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/marshmallow/schema.py", line 125, in __new__
dict_cls=dict_cls,
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py", line 94, in get_declared_fields
fields.update(mcs.get_auto_fields(fields, converter, opts, dict_cls))
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py", line 108, in get_auto_fields
for field_name, field in fields.items()
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py", line 110, in <dictcomp>
and field_name not in opts.exclude
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/marshmallow_sqlalchemy/schema/sqlalchemy_schema.py", line 28, in create_field
return converter.field_for(model, column_name, **self.field_kwargs)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/marshmallow_sqlalchemy/convert.py", line 171, in field_for
return self.property2field(prop, **kwargs)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/marshmallow_sqlalchemy/convert.py", line 146, in property2field
field_class = field_class or self._get_field_class_for_property(prop)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/marshmallow_sqlalchemy/convert.py", line 210, in _get_field_class_for_property
column = prop.columns[0]
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 1220, in __getattr__
return self._fallback_getattr(key)
File "/Users/jorzhang/tmp2/venv/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 1194, in _fallback_getattr
raise AttributeError(key)
AttributeError: columns
```
**What you expected to happen**:
I expect to be able to init the DB and run the webserver, but neither of these operations work.
**How to reproduce it**:
As above, a fresh install of airflow with python 3.7.10 (have not tried this for other python versions) in a new venv will produce this problem.
**Anything else we need to know**:
I understand that airflow is meant to be installed alongside a constraints file, but this is currently not possible with poetry, as evidenced by [this](https://github.com/python-poetry/poetry/issues/3225) thread. However, in that same thread @potiuk describes constraints as a way to make sure that for any given airflow version, as long as the constraints are respect the user should be able to install any set of extras and have their dependent library be conflict free.
The problem is: I expect that the base installation without constraints file should be fully functional (i.e. I am able to use the base library, plus the scheduler and webserver at least locally) as long as I don't install any extras, but it is not. Please let me know if this is not a valid assumption!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15447: `db init` and `webserver` commands do not work on fresh install without constraints
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15447:
URL: https://github.com/apache/airflow/issues/15447#issuecomment-823046023
Unfortunately we are not able to guarantee that released versions of Airflow will continue to install without constraints in the future.
The reasons why we have chosen constraints approach and why Airflow is different than many other applications and libraries are explained in detail in this chapter https://github.com/apache/airflow#installing-from-pypi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #15447: `db init` and `webserver` commands do not work on fresh install without constraints
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #15447:
URL: https://github.com/apache/airflow/issues/15447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] AlienationX commented on issue #15447: `db init` and `webserver` commands do not work on fresh install without constraints
Posted by GitBox <gi...@apache.org>.
AlienationX commented on issue #15447:
URL: https://github.com/apache/airflow/issues/15447#issuecomment-825524318
> @AlienationX you can include the needed constraint dependencies by doing
>
> ```
> pip install apache-airflow==2.0.1 --constraint <constraint file for your env>
> ```
>
> as described [here](https://airflow.apache.org/docs/apache-airflow/stable/start/local.html).
>
> If you're using poetry, you can do `poetry run pip ...` followed by the same command instead.
It's ok, thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #15447: `db init` and `webserver` commands do not work on fresh install without constraints
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #15447:
URL: https://github.com/apache/airflow/issues/15447#issuecomment-822983032
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zhzhang commented on issue #15447: `db init` and `webserver` commands do not work on fresh install without constraints
Posted by GitBox <gi...@apache.org>.
zhzhang commented on issue #15447:
URL: https://github.com/apache/airflow/issues/15447#issuecomment-824229341
@AlienationX you can include the needed constraint dependencies by doing
```
pip install apache-airflow==2.0.1 --constraint <constraint file for your env>
```
as described [here](https://airflow.apache.org/docs/apache-airflow/stable/start/local.html).
If you're using poetry, you can do `poetry run pip ...` followed by the same command instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] AlienationX commented on issue #15447: `db init` and `webserver` commands do not work on fresh install without constraints
Posted by GitBox <gi...@apache.org>.
AlienationX commented on issue #15447:
URL: https://github.com/apache/airflow/issues/15447#issuecomment-823743026
I had the same problem, how to solve it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org