You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/19 09:46:30 UTC
[GitHub] [airflow] ibardarov-fms opened a new issue, #28458: Upgrade to 2.5.0 fails on creating task_instance_note table
ibardarov-fms opened a new issue, #28458:
URL: https://github.com/apache/airflow/issues/28458
### Apache Airflow version
2.5.0
### What happened
I am upgrading from 2.4.1 to 2.50.
When doing the airflow db upgrade I got
```
just airflow-db-upgrade
./airflow.sh db upgrade
[+] Running 2/0
⠿ Container airflow-redis-1 Running 0.0s
⠿ Container airflow-airflow-init-1 Created 0.0s
[+] Running 2/2
⠿ Container airflow-redis-1 Healthy 1.2s
⠿ Container airflow-airflow-init-1 Started 1.6s
DB: postgresql+psycopg2://airflow_us_east_1:***@analytics-postgres-14-4.cluster-chemdf9ueboq.us-east-1.rds.amazonaws.com:5432/airflow_us_east_1
Performing upgrade with database postgresql+psycopg2://airflow_us_east_1:***@analytics-postgres-14-4.cluster-chemdf9ueboq.us-east-1.rds.amazonaws.com:5432/airflow_us_east_1
[2022-12-19 09:22:53,278] {migration.py:204} INFO - Context impl PostgresqlImpl.
[2022-12-19 09:22:53,279] {migration.py:211} INFO - Will assume transactional DDL.
[2022-12-19 09:22:53,403] {db.py:1570} INFO - Creating tables
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade ee8d93fcc81e -> 1986afd32c1b, Add DagRunNote and TaskInstanceNote
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
cursor, statement, parameters, context
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.InvalidForeignKey: there is no unique constraint matching given keys for referenced table "task_instance"
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 39, in main
args.func(args)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 52, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 108, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/db_command.py", line 88, in upgradedb
reserialize_dags=args.reserialize_dags,
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 75, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/db.py", line 1577, in upgradedb
command.upgrade(config, revision=to_revision or "heads")
File "/home/airflow/.local/lib/python3.7/site-packages/alembic/command.py", line 322, in upgrade
script.run_env()
File "/home/airflow/.local/lib/python3.7/site-packages/alembic/script/base.py", line 569, in run_env
util.load_python_file(self.dir, "env.py")
File "/home/airflow/.local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 94, in load_python_file
module = load_module_py(module_id, path)
File "/home/airflow/.local/lib/python3.7/site-packages/alembic/util/pyfiles.py", line 110, in load_module_py
spec.loader.exec_module(module) # type: ignore
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/migrations/env.py", line 117, in <module>
run_migrations_online()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/migrations/env.py", line 111, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/home/airflow/.local/lib/python3.7/site-packages/alembic/runtime/environment.py", line 853, in run_migrations
self.get_context().run_migrations(**kw)
File "/home/airflow/.local/lib/python3.7/site-packages/alembic/runtime/migration.py", line 623, in run_migrations
step.migration_fn(**kw)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/migrations/versions/0121_2_5_0_add_dagrunnote_and_taskinstancenote.py", line 87, in upgrade
sa.ForeignKeyConstraint(("user_id",), ["ab_user.id"], name="task_instance_note_user_fkey"),
File "<string>", line 8, in create_table
File "<string>", line 3, in create_table
File "/home/airflow/.local/lib/python3.7/site-packages/alembic/operations/ops.py", line 1254, in create_table
return operations.invoke(op)
File "/home/airflow/.local/lib/python3.7/site-packages/alembic/operations/base.py", line 399, in invoke
return fn(self, operation)
File "/home/airflow/.local/lib/python3.7/site-packages/alembic/operations/toimpl.py", line 114, in create_table
operations.impl.create_table(table)
File "/home/airflow/.local/lib/python3.7/site-packages/alembic/ddl/impl.py", line 354, in create_table
self._exec(schema.CreateTable(table))
File "/home/airflow/.local/lib/python3.7/site-packages/alembic/ddl/impl.py", line 195, in _exec
return conn.execute(construct, multiparams)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1380, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 81, in _execute_on_connection
self, multiparams, params, execution_options
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1478, in _execute_ddl
compiled,
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1944, in _execute_context
e, statement, parameters, cursor, context
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2125, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 210, in raise_
raise exception
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1901, in _execute_context
cursor, statement, parameters, context
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.InvalidForeignKey) there is no unique constraint matching given keys for referenced table "task_instance"
[SQL:
CREATE TABLE task_instance_note (
user_id INTEGER,
task_id VARCHAR(250) NOT NULL,
dag_id VARCHAR(250) NOT NULL,
run_id VARCHAR(250) NOT NULL,
map_index INTEGER NOT NULL,
content VARCHAR(1000),
created_at TIMESTAMP WITH TIME ZONE NOT NULL,
updated_at TIMESTAMP WITH TIME ZONE NOT NULL,
CONSTRAINT task_instance_note_pkey PRIMARY KEY (task_id, dag_id, run_id, map_index),
CONSTRAINT task_instance_note_ti_fkey FOREIGN KEY(dag_id, task_id, run_id, map_index) REFERENCES task_instance (dag_id, task_id, run_id, map_index) ON DELETE CASCADE,
CONSTRAINT task_instance_note_user_fkey FOREIGN KEY(user_id) REFERENCES ab_user (id)
)
]
(Background on this error at: https://sqlalche.me/e/14/f405)
error: Recipe `airflow-db-upgrade` failed on line 11 with exit code 1
```
### What you think should happen instead
I would expect the task_instance table to have all the keys created for me by previous migrations.
### How to reproduce
I don't know - maybe try to upgrade from 2.4.1 to 2.50 using postgres as a storage.
### Operating System
Linux
### Versions of Apache Airflow Providers
not related.
### Deployment
Docker-Compose
### Deployment details
The postgres is serverless "Aurora PostgreSQL"
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ibardarov-fms commented on issue #28458: Upgrade to 2.5.0 fails on creating task_instance_note table
Posted by GitBox <gi...@apache.org>.
ibardarov-fms commented on issue #28458:
URL: https://github.com/apache/airflow/issues/28458#issuecomment-1357879311
Here it is
```
airflow_us_east_1=> \d+ task_instance
Table "public.task_instance"
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
----------------------+-----------------------------+-----------+----------+---------------+----------+-------------+--------------+-------------
task_id | character varying(250) | | not null | | extended | | |
dag_id | character varying(250) | | not null | | extended | | |
run_id | character varying(250) | | not null | | extended | | |
start_date | timestamp with time zone | | | | plain | | |
end_date | timestamp with time zone | | | | plain | | |
duration | double precision | | | | plain | | |
state | character varying(20) | | | | extended | | |
try_number | integer | | | | plain | | |
hostname | character varying(1000) | | | | extended | | |
unixname | character varying(1000) | | | | extended | | |
job_id | integer | | | | plain | | |
pool | character varying(256) | | not null | | extended | | |
queue | character varying(256) | | | | extended | | |
priority_weight | integer | | | | plain | | |
operator | character varying(1000) | | | | extended | | |
queued_dttm | timestamp with time zone | | | | plain | | |
pid | integer | | | | plain | | |
max_tries | integer | | | '-1'::integer | plain | | |
executor_config | bytea | | | | extended | | |
pool_slots | integer | | not null | | plain | | |
queued_by_job_id | integer | | | | plain | | |
external_executor_id | character varying(250) | | | | extended | | |
trigger_id | integer | | | | plain | | |
trigger_timeout | timestamp without time zone | | | | plain | | |
next_method | character varying(1000) | | | | extended | | |
next_kwargs | json | | | | extended | | |
map_index | integer | | not null | '-1'::integer | plain | | |
updated_at | timestamp with time zone | | | | plain | | |
```
After I added the migration was successful, but I don't know whether I have added the correct index.
```
CREATE UNIQUE INDEX task_instance_unique_index ON task_instance (dag_id, task_id, run_id, map_index);
```
Now it looks
```
airflow_us_east_1=> \d+ task_instance
Table "public.task_instance"
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
----------------------+-----------------------------+-----------+----------+---------------+----------+-------------+--------------+-------------
task_id | character varying(250) | | not null | | extended | | |
dag_id | character varying(250) | | not null | | extended | | |
run_id | character varying(250) | | not null | | extended | | |
start_date | timestamp with time zone | | | | plain | | |
end_date | timestamp with time zone | | | | plain | | |
duration | double precision | | | | plain | | |
state | character varying(20) | | | | extended | | |
try_number | integer | | | | plain | | |
hostname | character varying(1000) | | | | extended | | |
unixname | character varying(1000) | | | | extended | | |
job_id | integer | | | | plain | | |
pool | character varying(256) | | not null | | extended | | |
queue | character varying(256) | | | | extended | | |
priority_weight | integer | | | | plain | | |
operator | character varying(1000) | | | | extended | | |
queued_dttm | timestamp with time zone | | | | plain | | |
pid | integer | | | | plain | | |
max_tries | integer | | | '-1'::integer | plain | | |
executor_config | bytea | | | | extended | | |
pool_slots | integer | | not null | | plain | | |
queued_by_job_id | integer | | | | plain | | |
external_executor_id | character varying(250) | | | | extended | | |
trigger_id | integer | | | | plain | | |
trigger_timeout | timestamp without time zone | | | | plain | | |
next_method | character varying(1000) | | | | extended | | |
next_kwargs | json | | | | extended | | |
map_index | integer | | not null | '-1'::integer | plain | | |
updated_at | timestamp with time zone | | | | plain | | |
Indexes:
"task_instance_unique_index" UNIQUE, btree (dag_id, task_id, run_id, map_index)
Referenced by:
TABLE "task_instance_note" CONSTRAINT "task_instance_note_ti_fkey" FOREIGN KEY (dag_id, task_id, run_id, map_index) REFERENCES task_instance(dag_id, task_id, run_id, map_index) ON DELETE CASCADE
Access method: heap
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ibardarov-fms commented on issue #28458: Upgrade to 2.5.0 fails on creating task_instance_note table
Posted by GitBox <gi...@apache.org>.
ibardarov-fms commented on issue #28458:
URL: https://github.com/apache/airflow/issues/28458#issuecomment-1357911566
I did a brand new installation on 2.5.0 and it looks that the indexes are there.
```
Table "public.task_instance"
Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description
----------------------+-----------------------------+-----------+----------+---------------+----------+-------------+--------------+-------------
task_id | character varying(250) | | not null | | extended | | |
dag_id | character varying(250) | | not null | | extended | | |
run_id | character varying(250) | | not null | | extended | | |
map_index | integer | | not null | '-1'::integer | plain | | |
start_date | timestamp with time zone | | | | plain | | |
end_date | timestamp with time zone | | | | plain | | |
duration | double precision | | | | plain | | |
state | character varying(20) | | | | extended | | |
try_number | integer | | | | plain | | |
max_tries | integer | | | '-1'::integer | plain | | |
hostname | character varying(1000) | | | | extended | | |
unixname | character varying(1000) | | | | extended | | |
job_id | integer | | | | plain | | |
pool | character varying(256) | | not null | | extended | | |
pool_slots | integer | | not null | | plain | | |
queue | character varying(256) | | | | extended | | |
priority_weight | integer | | | | plain | | |
operator | character varying(1000) | | | | extended | | |
queued_dttm | timestamp with time zone | | | | plain | | |
queued_by_job_id | integer | | | | plain | | |
pid | integer | | | | plain | | |
executor_config | bytea | | | | extended | | |
updated_at | timestamp with time zone | | | | plain | | |
external_executor_id | character varying(250) | | | | extended | | |
trigger_id | integer | | | | plain | | |
trigger_timeout | timestamp without time zone | | | | plain | | |
next_method | character varying(1000) | | | | extended | | |
next_kwargs | json | | | | extended | | |
Indexes:
"task_instance_pkey" PRIMARY KEY, btree (dag_id, task_id, run_id, map_index)
"ti_dag_run" btree (dag_id, run_id)
"ti_dag_state" btree (dag_id, state)
"ti_job_id" btree (job_id)
"ti_pool" btree (pool, state, priority_weight)
"ti_state" btree (state)
"ti_state_lkp" btree (dag_id, task_id, run_id, state)
"ti_trigger_id" btree (trigger_id)
Foreign-key constraints:
"task_instance_dag_run_fkey" FOREIGN KEY (dag_id, run_id) REFERENCES dag_run(dag_id, run_id) ON DELETE CASCADE
"task_instance_trigger_id_fkey" FOREIGN KEY (trigger_id) REFERENCES trigger(id) ON DELETE CASCADE
Referenced by:
TABLE "rendered_task_instance_fields" CONSTRAINT "rtif_ti_fkey" FOREIGN KEY (dag_id, task_id, run_id, map_index) REFERENCES task_instance(dag_id, task_id, run_id, map_index) ON DELETE CASCADE
TABLE "task_fail" CONSTRAINT "task_fail_ti_fkey" FOREIGN KEY (dag_id, task_id, run_id, map_index) REFERENCES task_instance(dag_id, task_id, run_id, map_index) ON DELETE CASCADE
TABLE "task_instance_note" CONSTRAINT "task_instance_note_ti_fkey" FOREIGN KEY (dag_id, task_id, run_id, map_index) REFERENCES task_instance(dag_id, task_id, run_id, map_index) ON DELETE CASCADE
TABLE "task_map" CONSTRAINT "task_map_task_instance_fkey" FOREIGN KEY (dag_id, task_id, run_id, map_index) REFERENCES task_instance(dag_id, task_id, run_id, map_index) ON DELETE CASCADE
TABLE "task_reschedule" CONSTRAINT "task_reschedule_ti_fkey" FOREIGN KEY (dag_id, task_id, run_id, map_index) REFERENCES task_instance(dag_id, task_id, run_id, map_index) ON DELETE CASCADE
TABLE "xcom" CONSTRAINT "xcom_task_instance_fkey" FOREIGN KEY (dag_id, task_id, run_id, map_index) REFERENCES task_instance(dag_id, task_id, run_id, map_index) ON DELETE CASCADE
Access method: heap
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #28458: Upgrade to 2.5.0 fails on creating task_instance_note table
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #28458:
URL: https://github.com/apache/airflow/issues/28458#issuecomment-1357583320
Could you provide the schema dump of your `task_instance` table please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #28458: Upgrade to 2.5.0 fails on creating task_instance_note table
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #28458:
URL: https://github.com/apache/airflow/issues/28458#issuecomment-1357371021
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ibardarov-fms closed issue #28458: Upgrade to 2.5.0 fails on creating task_instance_note table
Posted by GitBox <gi...@apache.org>.
ibardarov-fms closed issue #28458: Upgrade to 2.5.0 fails on creating task_instance_note table
URL: https://github.com/apache/airflow/issues/28458
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org