You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Jarek Potiuk (Jira)" <ji...@apache.org> on 2019/12/29 13:41:00 UTC
[jira] [Updated] (AIRFLOW-6265) Migration might behave wrongly when
DAGS parsed use DB
[ https://issues.apache.org/jira/browse/AIRFLOW-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jarek Potiuk updated AIRFLOW-6265:
----------------------------------
Affects Version/s: 2.0.0
> Migration might behave wrongly when DAGS parsed use DB
> ------------------------------------------------------
>
> Key: AIRFLOW-6265
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6265
> Project: Apache Airflow
> Issue Type: Bug
> Components: models
> Affects Versions: 1.10.2, 1.10.3, 1.10.4, 2.0.0, 1.10.5, 1.10.6
> Reporter: Jarek Potiuk
> Priority: Minor
>
> When running migratedb there might be a problem displayed during migration - in case some of the DAGs of the user use the DB in __init__ (for example reading connection table) they might fail mid-migration with errors for example the one below.
>
> The root cause is [https://github.com/apache/airflow/blob/master/airflow/migrations/versions/cc1e65623dc7_add_max_tries_column_to_task_instance.py#L70] - DagBag object creation which parses all the DAGs.
>
> It it is a minor annoyance - some of the TaskInstances can be not migrated properly with the max_tries not copied from task to task_instance.
> airflow | ========= ABOUT TO INIT DB ===========
> ...
> airflow | DB:
> [postgresql+psycopg2://airflow]
> :***@postgres:5432/airflow
> airflow | [2019-12-17 18:17:06,471] \{db.py:350} INFO - Creating tables
> airflow | INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
> airflow | INFO [alembic.runtime.migration] Will assume transactional DDL.
> airflow | INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current schema
> airflow | INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
> airflow | INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 13eb55f81627, maintain history for compatibility with earlier migrations
> ...
> airflow_db | 2019-12-17 18:17:07.001 UTC [55] ERROR: relation "connection" does not exist at character 481
> airflow_db | 2019-12-17 18:17:07.001 UTC [55] STATEMENT: SELECT connection.password AS connection_password, connection.extra AS connection_extra, connection.id AS connection_id, connection.conn_id AS connection_conn_id, connection.conn_type AS connection_conn_type, connection.host AS connection_host, connection.schema AS connection_schema, connection.login AS connection_login, connection.port AS connection_port, connection.is_encrypted AS connection_is_encrypted, connection.is_extra_encrypted AS connection_is_extra_encrypted
> airflow_db | FROM connection
> airflow_db | WHERE connection.conn_id = 'redshift_test_connection'
> airflow | ERROR [airflow.models.DagBag] Failed to import: /airflow/repo/dags/papermill_test/test_papermill_connection_injection/papermill_dag.py
> airflow | Traceback (most recent call last):
> airflow | File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
> airflow | cursor, statement, parameters, context
> airflow | File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute
> airflow | cursor.execute(statement, parameters)
> airflow | psycopg2.errors.UndefinedTable: relation "connection" does not exist
> airflow | LINE 2: FROM connection
> ...
> airflow | INFO [alembic.runtime.migration] Running upgrade cc1e65623dc7 -> bdaa763e6c56, Make xcom value column a large binary
> airflow | INFO [alembic.runtime.migration] Running upgrade bdaa763e6c56 -> 947454bf1dff, add ti job_id index
> airflow | INFO [alembic.runtime.migration] Running upgrade 947454bf1dff -> d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text types)
> airflow | INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 0e2a74e0fc9f, Add time zone awareness
> airflow | INFO [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 33ae817a1ff4, kubernetes_resource_checkpointing
> airflow | INFO [alembic.runtime.migration] Running upgrade 33ae817a1ff4 -> 27c6a30d7c24, kubernetes_resource_checkpointing
> ...
> ========= SLEEPING ===========
--
This message was sent by Atlassian Jira
(v8.3.4#803005)