You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/06 14:33:35 UTC
[GitHub] [airflow] RafayAK opened a new issue #15230: New version of sqlalchemy is breaking Airflow installation.
RafayAK opened a new issue #15230:
URL: https://github.com/apache/airflow/issues/15230
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**:
2.0.1
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
none
**Environment**:
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- ubuntu 20.04
- **Kernel** (e.g. `uname -a`):
Linux rafay-Inspiron-7537 5.8.0-48-generic #54~20.04.1-Ubuntu SMP Sat Mar 20 13:40:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
- **Install tools**:
- **Others**:
**What happened**:
```bash
[2021-04-06 10:58:25,430] {db.py:674} INFO - Creating tables
/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/relationships.py:3463 SAWarning: relationship 'DagRun.serialized_dag' will copy column serialized_dag.dag_id to column dag_run.dag_id, which conflicts with relationship(s): 'TaskInstance.dag_run' (copies task_instance.dag_id to dag_run.dag_id), 'DagRun.task_instances' (copies task_instance.dag_id to dag_run.dag_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards. The 'overlaps' parameter may be used to remove this warning. (Background on this error at: http://sqlalche.me/e/14/qzyx)
/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/relationships.py:3463 SAWarning: relationship 'SerializedDagModel.dag_runs' will copy column serialized_dag.dag_id to column dag_run.dag_id, which conflicts with relationship(s): 'TaskInstance.dag_run' (copies task_instance.dag_id to dag_run.dag_id), 'DagRun.task_instances' (copies task_instance.dag_id to dag_run.dag_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards. The 'overlaps' parameter may be used to remove this warning. (Background on this error at: http://sqlalche.me/e/14/qzyx)
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/usr/local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/db_command.py", line 31, in initdb
db.initdb()
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 549, in initdb
upgradedb()
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 684, in upgradedb
command.upgrade(config, 'heads')
File "/usr/local/lib/python3.7/site-packages/alembic/command.py", line 294, in upgrade
script.run_env()
File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 490, in run_env
util.load_python_file(self.dir, "env.py")
File "/usr/local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.7/site-packages/alembic/util/compat.py", line 182, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.7/site-packages/airflow/migrations/env.py", line 108, in <module>
run_migrations_online()
File "/usr/local/lib/python3.7/site-packages/airflow/migrations/env.py", line 102, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/usr/local/lib/python3.7/site-packages/alembic/runtime/environment.py", line 813, in run_migrations
self.get_context().run_migrations(**kw)
File "/usr/local/lib/python3.7/site-packages/alembic/runtime/migration.py", line 548, in run_migrations
for step in self._migrations_fn(heads, self):
File "/usr/local/lib/python3.7/site-packages/alembic/command.py", line 283, in upgrade
return script._upgrade_revs(revision, rev)
File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 365, in _upgrade_revs
revs = list(revs)
File "/usr/local/lib/python3.7/site-packages/alembic/script/revision.py", line 904, in _iterate_revisions
requested_lowers = self.get_revisions(lower)
File "/usr/local/lib/python3.7/site-packages/alembic/script/revision.py", line 455, in get_revisions
return sum([self.get_revisions(id_elem) for id_elem in id_], ())
File "/usr/local/lib/python3.7/site-packages/alembic/script/revision.py", line 455, in <listcomp>
return sum([self.get_revisions(id_elem) for id_elem in id_], ())
File "/usr/local/lib/python3.7/site-packages/alembic/script/revision.py", line 457, in get_revisions
resolved_id, branch_label = self._resolve_revision_number(id_)
File "/usr/local/lib/python3.7/site-packages/alembic/script/revision.py", line 640, in _resolve_revision_number
self._revision_map
File "/usr/local/lib/python3.7/site-packages/alembic/util/langhelpers.py", line 234, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "/usr/local/lib/python3.7/site-packages/alembic/script/revision.py", line 156, in _revision_map
for revision in self._generator():
File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 115, in _load_revisions
script = Script._from_filename(self, vers, file_)
File "/usr/local/lib/python3.7/site-packages/alembic/script/base.py", line 904, in _from_filename
module = util.load_python_file(dir_, filename)
File "/usr/local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
module = load_module_py(module_id, path)
File "/usr/local/lib/python3.7/site-packages/alembic/util/compat.py", line 182, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.7/site-packages/airflow/migrations/versions/82b7c48c147f_remove_can_read_permission_on_config_.py", line 28, in <module>
from airflow.www.app import create_app
File "/usr/local/lib/python3.7/site-packages/airflow/www/app.py", line 38, in <module>
from airflow.www.extensions.init_views import (
File "/usr/local/lib/python3.7/site-packages/airflow/www/extensions/init_views.py", line 29, in <module>
from airflow.www.views import lazy_add_provider_discovered_options_to_connection_form
File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 96, in <module>
from airflow.www import auth, utils as wwwutils
File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line 27, in <module>
from flask_appbuilder.models.sqla.interface import SQLAInterface
File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/models/sqla/interface.py", line 16, in <module>
from sqlalchemy_utils.types.uuid import UUIDType
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/__init__.py", line 1, in <module>
from .aggregates import aggregated # noqa
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/aggregates.py", line 372, in <module>
from .functions.orm import get_column_key
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/functions/__init__.py", line 1, in <module>
from .database import ( # noqa
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/functions/database.py", line 11, in <module>
from .orm import quote
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/functions/orm.py", line 14, in <module>
from sqlalchemy.orm.query import _ColumnEntity
ImportError: cannot import name '_ColumnEntity' from 'sqlalchemy.orm.query' (/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py)
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/usr/local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 47, in command
func = import_string(import_path)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/module_loading.py", line 32, in import_string
module = import_module(module_path)
File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/user_command.py", line 29, in <module>
from airflow.www.app import cached_app
File "/usr/local/lib/python3.7/site-packages/airflow/www/app.py", line 38, in <module>
from airflow.www.extensions.init_views import (
File "/usr/local/lib/python3.7/site-packages/airflow/www/extensions/init_views.py", line 29, in <module>
from airflow.www.views import lazy_add_provider_discovered_options_to_connection_form
File "/usr/local/lib/python3.7/site-packages/airflow/www/views.py", line 96, in <module>
from airflow.www import auth, utils as wwwutils
File "/usr/local/lib/python3.7/site-packages/airflow/www/utils.py", line 27, in <module>
from flask_appbuilder.models.sqla.interface import SQLAInterface
File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/models/sqla/interface.py", line 16, in <module>
from sqlalchemy_utils.types.uuid import UUIDType
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/__init__.py", line 1, in <module>
from .aggregates import aggregated # noqa
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/aggregates.py", line 372, in <module>
from .functions.orm import get_column_key
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/functions/__init__.py", line 1, in <module>
from .database import ( # noqa
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/functions/database.py", line 11, in <module>
from .orm import quote
File "/usr/local/lib/python3.7/site-packages/sqlalchemy_utils/functions/orm.py", line 14, in <module>
from sqlalchemy.orm.query import _ColumnEntity
ImportError: cannot import name '_ColumnEntity' from 'sqlalchemy.orm.query' (/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/query.py)
```
**What you expected to happen**:
`airflow db init` should execute without a hitch
<!-- What do you think went wrong? -->
Sqlalchemy package has been updated with some breaking changes. pip is installing `sqlalchemy-1.4.5`.
https://stackoverflow.com/questions/66644975/importerror-cannot-import-name-columnentity-from-sqlalchemy-orm-query
**How to reproduce it**:
` pip install apache-airflow[slack,statsd] `
**Anything else we need to know**:
<!--
How often does this problem occur? Once? Every time etc?
Always through pip now.
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #15230: New version of sqlalchemy is breaking Airflow installation.
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #15230:
URL: https://github.com/apache/airflow/issues/15230
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15230: New version of sqlalchemy is breaking Airflow installation.
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15230:
URL: https://github.com/apache/airflow/issues/15230#issuecomment-814236689
Upstream bug entry: kvesteri/sqlalchemy-utils#505
Note that it is highly recommended to install Airflow with the constraints files, instead of a plain `pip install`. This ensures the dependency set passes the test suite and is promised to work. More instructions are available in the documentation: https://airflow.apache.org/docs/apache-airflow/stable/installation.html#constraints-files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #15230: New version of sqlalchemy is breaking Airflow installation.
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #15230:
URL: https://github.com/apache/airflow/issues/15230#issuecomment-814268829
Hey yes like @uranusjr suggested please use the constraints file as Airflow is a library and an application, the upstream dependency can time and again break hence we keep set of files that work for sure.
We had fixed sqlalchmey version in https://github.com/apache/airflow/commit/c29f6fb76b9d87c50713ae94fda805b9f789a01d#diff-fa602a8a75dc9dcc92261bac5f533c2a85e34fcceaff63b3a3a81d9acde2fc52
So I will mark this issue as a duplicate of #14811 . Please let us know if you have any issues using the constraints file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org