You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/30 03:57:32 UTC
[GitHub] [airflow] ChandelAnushka opened a new issue, #27998: Airflow Database Upgrade error: sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "operator" of relation "task_instance" already exists
ChandelAnushka opened a new issue, #27998:
URL: https://github.com/apache/airflow/issues/27998
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
Apache Airflow version
2.3.2
What happened
My production deployment had Airflow v2.1.2. It already had Dags, Task Instances, etc., in the tables. When upgrading to Airflow v2.3.2, the execution of airflow db upgrade resulted in the failure: sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "operator" of relation "task_instance" already exists
The logs/traceback:
```
[2022-11-29 20:39:35,050] {{db.py:1448}} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade 13eb55f81627 -> 338e90f54d61, Add ``operator`` and ``queued_dttm`` to ``task_instance`` table
Traceback (most recent call last):
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1706, in _execute_context
cursor, statement, parameters, context
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 716, in do_execute
cursor.execute(statement, parameters)
psycopg2.ProgrammingError: column "operator" of relation "task_instance" already exists
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/bin/airflow", line 8, in <module>
sys.exit(main())
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/airflow/__main__.py", line 38, in main
args.func(args)
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 51, in command
return func(*args, **kwargs)
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/airflow/utils/cli.py", line 99, in wrapper
return f(*args, **kwargs)
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/airflow/cli/commands/db_command.py", line 82, in upgradedb
db.upgradedb(to_revision=to_revision, from_revision=from_revision, show_sql_only=args.show_sql_only)
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/airflow/utils/session.py", line 71, in wrapper
return func(*args, session=session, **kwargs)
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/airflow/utils/db.py", line 1449, in upgradedb
command.upgrade(config, revision=to_revision or 'heads')
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/alembic/command.py", line 320, in upgrade
script.run_env()
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/alembic/script/base.py", line 563, in run_env
util.load_python_file(self.dir, "env.py")
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
module = load_module_py(module_id, path)
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/airflow/migrations/env.py", line 107, in <module>
run_migrations_online()
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/airflow/migrations/env.py", line 101, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
self.get_context().run_migrations(**kw)
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/alembic/runtime/migration.py", line 612, in run_migrations
step.migration_fn(**kw)
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/airflow/migrations/versions/0004_1_5_0_more_logging_into_task_isntance.py", line 38, in upgrade
op.add_column('task_instance', sa.Column('operator', sa.String(length=1000), nullable=True))
File "<string>", line 8, in add_column
File "<string>", line 3, in add_column
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/alembic/operations/ops.py", line 2047, in add_column
return operations.invoke(op)
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/alembic/operations/base.py", line 387, in invoke
return fn(self, operation)
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/alembic/operations/toimpl.py", line 154, in add_column
operations.impl.add_column(table_name, column, schema=schema, **kw)
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/alembic/ddl/impl.py", line 324, in add_column
self._exec(base.AddColumn(table_name, column, schema=schema))
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/alembic/ddl/impl.py", line 197, in _exec
return conn.execute(construct, multiparams)
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1200, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 78, in _execute_on_connection
self, multiparams, params, execution_options
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1296, in _execute_ddl
compiled,
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1749, in _execute_context
e, statement, parameters, cursor, context
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1930, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
raise exception
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1706, in _execute_context
cursor, statement, parameters, context
File "/amex/mlp/airflow2_celery/airflow_env_2022_09_29/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 716, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "operator" of relation "task_instance" already exists
[SQL: ALTER TABLE task_instance ADD COLUMN operator VARCHAR(1000)]
```
### What you think should happen instead
The database upgrade should have succeeded.
### How to reproduce
Install Airflow 2.1.0 and initialize airflow db.
Create and execute some dags so that Dag table gets populated.
Upgrade airflow to 2.3.2 and execute airflow db upgrade.
### Operating System
Mac OS Monterey
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon 6.0.0
apache-airflow-providers-celery 3.0.0
apache-airflow-providers-ftp 3.1.0
apache-airflow-providers-imap 3.0.0
apache-airflow-providers-papermill 3.0.0
apache-airflow-providers-postgres 5.2.0
apache-airflow-providers-redis 3.0.0
apache-airflow-providers-sqlite 3.2.1
apache-airflow-providers-ssh 3.2.0
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27998: Airflow Database Upgrade error: sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "operator" of relation "task_instance" already exists
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27998:
URL: https://github.com/apache/airflow/issues/27998#issuecomment-1336650455
Upgrade to a newer version (even 2.5.0). There is no need to upgrade to 2.3.2 when you can upgrade to 2.5.0 and I think similar problem was fixed in one of the later versions. Try with 2.5.0 and see if it is still there.
Also check if the alembic revision is good for your version: https://airflow.apache.org/docs/apache-airflow/stable/migrations-ref.html
Converting to discussion for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #27998: Airflow Database Upgrade error: sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "operator" of relation "task_instance" already exists
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #27998: Airflow Database Upgrade error: sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "operator" of relation "task_instance" already exists
URL: https://github.com/apache/airflow/issues/27998
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tirkarthi commented on issue #27998: Airflow Database Upgrade error: sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) column "operator" of relation "task_instance" already exists
Posted by GitBox <gi...@apache.org>.
tirkarthi commented on issue #27998:
URL: https://github.com/apache/airflow/issues/27998#issuecomment-1335467007
See also similar discussion at https://github.com/apache/airflow/discussions/21803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org