You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Jarek Potiuk (Jira)" <ji...@apache.org> on 2019/12/17 19:05:00 UTC
[jira] [Created] (AIRFLOW-6265) Migration might behave wrongly when
DAGS parsed use DB
Jarek Potiuk created AIRFLOW-6265:
-------------------------------------
Summary: Migration might behave wrongly when DAGS parsed use DB
Key: AIRFLOW-6265
URL: https://issues.apache.org/jira/browse/AIRFLOW-6265
Project: Apache Airflow
Issue Type: Bug
Components: models
Affects Versions: 1.10.6, 1.10.5, 1.10.4, 1.10.3, 1.10.2
Reporter: Jarek Potiuk
When running migratedb there might be a problem displayed during migration - in case some of the DAGs of the user use the DB in __init__ (for example reading connection table) they might fail mid-migration with errors for example the one below.
The root cause is [https://github.com/apache/airflow/blob/master/airflow/migrations/versions/cc1e65623dc7_add_max_tries_column_to_task_instance.py#L70] - DagBag object creation which parses all the DAGs.
It it is a minor annoyance - some of the TaskInstances can be not migrated properly with the max_tries not copied from task to task_instance.
airflow | ========= ABOUT TO INIT DB ===========
...
airflow | DB:
[postgresql+psycopg2://airflow]
:***@postgres:5432/airflow
airflow | [2019-12-17 18:17:06,471] \{db.py:350} INFO - Creating tables
airflow | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
airflow | INFO [alembic.runtime.migration] Will assume transactional DDL.
airflow | INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current schema
airflow | INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
airflow | INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 13eb55f81627, maintain history for compatibility with earlier migrations
...
airflow_db | 2019-12-17 18:17:07.001 UTC [55] ERROR: relation "connection" does not exist at character 481
airflow_db | 2019-12-17 18:17:07.001 UTC [55] STATEMENT: SELECT connection.password AS connection_password, connection.extra AS connection_extra, connection.id AS connection_id, connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login, connection.port AS connection_port, connection.is_encrypted AS connection_is_encrypted, connection.is_extra_encrypted AS connection_is_extra_encrypted
airflow_db | FROM connection
airflow_db | WHERE connection.conn_id = 'redshift_test_connection'
airflow | ERROR [airflow.models.DagBag] Failed to import: /airflow/repo/dags/papermill_test/test_papermill_connection_injection/papermill_dag.py
airflow | Traceback (most recent call last):
airflow | File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
airflow | cursor, statement, parameters, context
airflow | File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
airflow | cursor.execute(statement, parameters)
airflow | psycopg2.errors.UndefinedTable: relation "connection" does not exist
airflow | LINE 2: FROM connection
...
airflow | INFO [alembic.runtime.migration] Running upgrade cc1e65623dc7 -> bdaa763e6c56, Make xcom value column a large binary
airflow | INFO [alembic.runtime.migration] Running upgrade bdaa763e6c56 -> 947454bf1dff, add ti job_id index
airflow | INFO [alembic.runtime.migration] Running upgrade 947454bf1dff -> d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text types)
airflow | INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 0e2a74e0fc9f, Add time zone awareness
airflow | INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 33ae817a1ff4, kubernetes_resource_checkpointing
airflow | INFO [alembic.runtime.migration] Running upgrade 33ae817a1ff4 -> 27c6a30d7c24, kubernetes_resource_checkpointing
...
========= SLEEPING ===========
--
This message was sent by Atlassian Jira
(v8.3.4#803005)