[jira] [Commented] (AIRFLOW-5036) Issue running resetdb with test config from code

["Marcin Mejran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-5036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16893864#comment-16893864 ] 

Marcin Mejran commented on AIRFLOW-5036:
----------------------------------------

Issues is caused by the Jobs class not being loaded by default anymore so SqlAlchemy isn't aware of the table so it's not dropped. This can be fixed by running:

import airflow.jobs

> Issue running resetdb with test config from code
> ------------------------------------------------
>
>                 Key: AIRFLOW-5036
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5036
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: database
>    Affects Versions: 1.10.4
>            Reporter: Marcin Mejran
>            Priority: Major
>
> In our local test of custom operators we've been doing an initdb and resetdb before each test. This worked fine in previous versions including 1.10.3. We recently tried the 1.10.4rc3 release and hit an issue.
> If I do an 'airflow initdb' and 'airflow resetdb' from command line it seems to work fine.
> If I run the following using Airflow 1.10.4rc3 on a clean db (ie: empty AIRFLOW_HOME).
>  
> {code:java}
> from airflow import configuration
> from airflow.utils import db
> configuration.load_test_config()
> db.initdb()
> db.resetdb(rbac=False)
> {code}
>  
> I get this error:
> {code:java}
> INFO [alembic.runtime.migration] Context impl SQLiteImpl.
> INFO [alembic.runtime.migration] Will assume non-transactional DDL.
> INFO [alembic.runtime.migration] Context impl SQLiteImpl.
> INFO [alembic.runtime.migration] Will assume non-transactional DDL.
> INFO [alembic.runtime.migration] Running upgrade -> e3a246e0dc1, current schema
> INFO [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
> alembic/ddl/sqlite.py:39: UserWarning: Skipping unsupported ALTER for creation of implicit constraint
> "Skipping unsupported ALTER for "
> INFO [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 13eb55f81627, maintain history for compatibility with earlier migrations
> INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 -> 338e90f54d61, More logging into task_instance
> INFO [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, job_id indices
> Traceback (most recent call last):
> File "sqlalchemy/engine/base.py", line 1244, in _execute_context
> cursor, statement, parameters, context
> File "sqlalchemy/engine/default.py", line 552, in do_execute
> cursor.execute(statement, parameters)
> sqlite3.OperationalError: index idx_job_state_heartbeat already exists
> The above exception was the direct cause of the following exception:
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "airflow/utils/db.py", line 406, in resetdb
> initdb(rbac)
> File "airflow/utils/db.py", line 106, in initdb
> upgradedb()
> File "airflow/utils/db.py", line 377, in upgradedb
> command.upgrade(config, 'heads')
> File "alembic/command.py", line 276, in upgrade
> script.run_env()
> File "alembic/script/base.py", line 475, in run_env
> util.load_python_file(self.dir, "env.py")
> File "alembic/util/pyfiles.py", line 90, in load_python_file
> module = load_module_py(module_id, path)
> File "alembic/util/compat.py", line 177, in load_module_py
> spec.loader.exec_module(module)
> File "<frozen importlib._bootstrap_external>", line 678, in exec_module
> File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
> File "airflow/migrations/env.py", line 92, in <module>
> run_migrations_online()
> File "airflow/migrations/env.py", line 86, in run_migrations_online
> context.run_migrations()
> File "<string>", line 8, in run_migrations
> File "alembic/runtime/environment.py", line 839, in run_migrations
> self.get_context().run_migrations(**kw)
> File "alembic/runtime/migration.py", line 362, in run_migrations
> step.migration_fn(**kw)
> File "airflow/migrations/versions/52d714495f0_job_id_indices.py", line 38, in upgrade
> ['state', 'latest_heartbeat'], unique=False)
> File "<string>", line 8, in create_index
> File "<string>", line 3, in create_index
> File "alembic/operations/ops.py", line 975, in create_index
> return operations.invoke(op)
> File "alembic/operations/base.py", line 345, in invoke
> return fn(self, operation)
> File "alembic/operations/toimpl.py", line 88, in create_index
> operations.impl.create_index(idx)
> File "alembic/ddl/impl.py", line 276, in create_index
> self._exec(schema.CreateIndex(index))
> File "alembic/ddl/impl.py", line 134, in _exec
> return conn.execute(construct, *multiparams, **params)
> File "sqlalchemy/engine/base.py", line 988, in execute
> return meth(self, multiparams, params)
> File "sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
> return connection._execute_ddl(self, multiparams, params)
> File "sqlalchemy/engine/base.py", line 1050, in _execute_ddl
> compiled,
> File "sqlalchemy/engine/base.py", line 1248, in _execute_context
> e, statement, parameters, cursor, context
> File "sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
> util.raise_from_cause(sqlalchemy_exception, exc_info)
> File "sqlalchemy/util/compat.py", line 398, in raise_from_cause
> reraise(type(exception), exception, tb=exc_tb, cause=cause)
> File "sqlalchemy/util/compat.py", line 152, in reraise
> raise value.with_traceback(tb)
> File "sqlalchemy/engine/base.py", line 1244, in _execute_context
> cursor, statement, parameters, context
> File "sqlalchemy/engine/default.py", line 552, in do_execute
> cursor.execute(statement, parameters)
> sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) index idx_job_state_heartbeat already exists
> [SQL: CREATE INDEX idx_job_state_heartbeat ON job (state, latest_heartbeat)]
> (Background on this error at: http://sqlalche.me/e/e3q8)
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)