You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/07 11:04:36 UTC
[GitHub] [airflow] george-zubrienko opened a new issue #15249: Error when initializing database for unit tests (sqllite)
george-zubrienko opened a new issue #15249:
URL: https://github.com/apache/airflow/issues/15249
**Apache Airflow version**: 2.0.1
**Environment**: Python virtual environment based on _circleci/python:3.7.9-stretch_
**What happened**: We have CircleCI running unit tests for an extensions library we built for airflow. It
It runs unit tests in a virtual environment initialized with the code below:
```python
python3 -m venv virtualenv
. virtualenv/bin/activate
pip install -U pip==19.3.1 wheel setuptools pylint
pip install -i https://$ARTIFACTORY_USERNAME:$ARTIFACTORY_PASSWORD@liveintent.jfrog.io/liveintent/api/pypi/pypi/simple/ -r requirements.txt
```
This was our `requirements.txt` file
```
pyparsing==2.4.6
pyrsistent==0.16.0
pytest==5.3.5
apache-airflow==2.0.1
apache-airflow-providers-amazon==1.1.0
apache-airflow-providers-cncf-kubernetes==1.0.1
apache-airflow-providers-postgres==1.0.1
apache-airflow-providers-sftp==1.1.0
apache-airflow-providers-ssh==1.1.0
psycopg2==2.8.6
alembic==1.4.3
PyJWT==1.7.1
importlib-metadata==1.7.0
```
We run unit tests and linting like this
```python
cd test
. ../virtualenv/bin/activate
export PYTHONPATH=$PYTHONPATH:/home/circleci/airflow-build/
airflow db init
pytest --junitxml pytest-result.xml
mkdir ../lint
cd ../li_airflow_common
find . -iname '*.py' | sort | xargs pylint -j0 --output-format=parseable --reports=n --msg-template='{path}:{module}:{line}: [{msg_id}({symbol}), {obj}] {msg}' | tee ../lint/pylint.log
```
As you can see, this will init a "local" airflow with SQLlite database. Recently we found that our builds that previously passed, fail with
```
DB: sqlite:////home/circleci/airflow/airflow.db
[2021-04-06 10:59:56,801] {db.py:674} INFO - Creating tables
/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/sqlalchemy/orm/relationships.py:3463 SAWarning: relationship 'DagRun.serialized_dag' will copy column serialized_dag.dag_id to column dag_run.dag_id, which conflicts with relationship(s): 'TaskInstance.dag_run' (copies task_instance.dag_id to dag_run.dag_id), 'DagRun.task_instances' (copies task_instance.dag_id to dag_run.dag_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards. The 'overlaps' parameter may be used to remove this warning. (Background on this error at: http://sqlalche.me/e/14/qzyx)
/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/sqlalchemy/orm/relationships.py:3463 SAWarning: relationship 'SerializedDagModel.dag_runs' will copy column serialized_dag.dag_id to column dag_run.dag_id, which conflicts with relationship(s): 'TaskInstance.dag_run' (copies task_instance.dag_id to dag_run.dag_id), 'DagRun.task_instances' (copies task_instance.dag_id to dag_run.dag_id). If this is not the intention, consider if these relationships should be linked with back_populates, or if viewonly=True should be applied to one or more if they are read-only. For the less common case that foreign key constraints are partially overlapping, the orm.foreign() annotation can be used to isolate the columns that should be written towards. The 'overlaps' parameter may be used to remove this warning. (Background on this error at: http://sqlalche.me/e/14/qzyx)
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
Traceback (most recent call last):
File "/home/circleci/airflow-build/virtualenv/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/airflow/cli/commands/db_command.py", line 31, in initdb
db.initdb()
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/airflow/utils/db.py", line 549, in initdb
upgradedb()
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/airflow/utils/db.py", line 684, in upgradedb
command.upgrade(config, 'heads')
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/command.py", line 298, in upgrade
script.run_env()
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/script/base.py", line 489, in run_env
util.load_python_file(self.dir, "env.py")
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
module = load_module_py(module_id, path)
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/util/compat.py", line 184, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/airflow/migrations/env.py", line 108, in <module>
run_migrations_online()
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/airflow/migrations/env.py", line 102, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/runtime/environment.py", line 846, in run_migrations
self.get_context().run_migrations(**kw)
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/runtime/migration.py", line 511, in run_migrations
for step in self._migrations_fn(heads, self):
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/command.py", line 287, in upgrade
return script._upgrade_revs(revision, rev)
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/script/base.py", line 364, in _upgrade_revs
revs = list(revs)
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/script/revision.py", line 777, in _iterate_revisions
uppers = util.dedupe_tuple(self.get_revisions(upper))
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/script/revision.py", line 321, in get_revisions
resolved_id, branch_label = self._resolve_revision_number(id_)
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/script/revision.py", line 501, in _resolve_revision_number
self._revision_map
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/util/langhelpers.py", line 230, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/script/revision.py", line 123, in _revision_map
for revision in self._generator():
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/script/base.py", line 112, in _load_revisions
script = Script._from_filename(self, vers, file_)
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/script/base.py", line 906, in _from_filename
module = util.load_python_file(dir_, filename)
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 98, in load_python_file
module = load_module_py(module_id, path)
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/alembic/util/compat.py", line 184, in load_module_py
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/airflow/migrations/versions/2c6edca13270_resource_based_permissions.py", line 29, in <module>
from airflow.www.app import create_app
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/airflow/www/app.py", line 38, in <module>
from airflow.www.extensions.init_views import (
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/airflow/www/extensions/init_views.py", line 29, in <module>
from airflow.www.views import lazy_add_provider_discovered_options_to_connection_form
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/airflow/www/views.py", line 96, in <module>
from airflow.www import auth, utils as wwwutils
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/airflow/www/utils.py", line 27, in <module>
from flask_appbuilder.models.sqla.interface import SQLAInterface
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/flask_appbuilder/models/sqla/interface.py", line 16, in <module>
from sqlalchemy_utils.types.uuid import UUIDType
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/sqlalchemy_utils/__init__.py", line 1, in <module>
from .aggregates import aggregated # noqa
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/sqlalchemy_utils/aggregates.py", line 372, in <module>
from .functions.orm import get_column_key
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/sqlalchemy_utils/functions/__init__.py", line 1, in <module>
from .database import ( # noqa
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/sqlalchemy_utils/functions/database.py", line 11, in <module>
from .orm import quote
File "/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/sqlalchemy_utils/functions/orm.py", line 14, in <module>
from sqlalchemy.orm.query import _ColumnEntity
ImportError: cannot import name '_ColumnEntity' from 'sqlalchemy.orm.query' (/home/circleci/airflow-build/virtualenv/lib/python3.7/site-packages/sqlalchemy/orm/query.py)
```
**What you expected to happen**: Builds pass with only airflow dependencies specified in `requirements.txt`. Quick googling points here https://stackoverflow.com/questions/66644975/importerror-cannot-import-name-columnentity-from-sqlalchemy-orm-query
**How to reproduce it**: Init a virtual env with `requirements.txt` contents from above
**Anything else we need to know**: This is fixed by adding `sqlalchemy < 1.4.0` to the `requirements.txt`. However, I feel this is more of a quick hack rather than a proper fix. I would expect `apache-airflow` to correctly reference its dependencies, so no amendments to imported versions are required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15249: Error when initializing database for unit tests (sqllite)
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15249:
URL: https://github.com/apache/airflow/issues/15249#issuecomment-814830661
Duplicate to #15230.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #15249: Error when initializing database for unit tests (sqllite)
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #15249:
URL: https://github.com/apache/airflow/issues/15249#issuecomment-815176622
Yep. You should use constraints. Closing as invalid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #15249: Error when initializing database for unit tests (sqllite)
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #15249:
URL: https://github.com/apache/airflow/issues/15249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr edited a comment on issue #15249: Error when initializing database for unit tests (sqllite)
Posted by GitBox <gi...@apache.org>.
uranusjr edited a comment on issue #15249:
URL: https://github.com/apache/airflow/issues/15249#issuecomment-814830661
Duplicate to #14811 and #15230.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org