You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Chais (via GitHub)" <gi...@apache.org> on 2025/11/03 14:16:18 UTC
[I] Downgrading DB leaves instance unable to clear tasks [airflow]
Chais opened a new issue, #57747:
URL: https://github.com/apache/airflow/issues/57747
### Apache Airflow version
2.11.0
### If "Other Airflow 2/3 version" selected, which one?
_No response_
### What happened?
Recently I accidentally applied DB migrations 28 through 68 while attempting to update from Airflow 2.10.5 to 2.11.0. After cleaning up the mess by downgrading the DB I noticed that I'm no longer able to clear tasks in order to re-try them.
Trying gives you nothing except a lengthy exception in the `webserver` logs, the relevant part of which is this:
<code>sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "id" of relation "task_instance_history" violates not-null constraint
DETAIL: Failing row contains (nosify.notify_successes, cert_env, manual__2025-11-02T23:00:04.866429+00:00, 0, 1, 2025-11-02 23:04:02.810671+00, 2025-11-02 23:04:05.093361+00, 2.28269, success,
0, airflow-worker-0.airflow-worker.airflow.svc.cluster.local, airflow, default_pool, 1, default, 1, _PythonDecoratedOperator, null, 2025-11-02 23:04:01.165684+00, 720119, 68415, null, \x80057d942e, 2025
-11-02 23:04:05.146802+00, null, a96e4277-0d0e-4b53-868d-48e72e2dd9bb, null, null, null, null, nosify.notify_successes, null, 721182).
[SQL: INSERT INTO task_instance_history (task_id, dag_id, run_id, map_index, try_number, start_date, end_date, duration, state, max_tries, hostname, unixname, job_id, pool, pool_slots, queue, p
riority_weight, operator, custom_operator_name, queued_dttm, queued_by_job_id, pid, executor, executor_config, updated_at, rendered_map_index, external_executor_id, trigger_id, trigger_timeout, next_meth
od, next_kwargs, task_display_name) VALUES (%(task_id)s, %(dag_id)s, %(run_id)s, %(map_index)s, %(try_number)s, %(start_date)s, %(end_date)s, %(duration)s, %(state)s, %(max_tries)s, %(hostname)s, %(unixn
ame)s, %(job_id)s, %(pool)s, %(pool_slots)s, %(queue)s, %(priority_weight)s, %(operator)s, %(custom_operator_name)s, %(queued_dttm)s, %(queued_by_job_id)s, %(pid)s, %(executor)s, %(executor_config)s, %(u
pdated_at)s, %(rendered_map_index)s, %(external_executor_id)s, %(trigger_id)s, %(trigger_timeout)s, %(next_method)s, %(next_kwargs)s, %(task_display_name)s) RETURNING task_instance_history.id]
[parameters: {'task_id': 'nosify.notify_successes', 'dag_id': 'cert_env', 'run_id': 'manual__2025-11-02T23:00:04.866429+00:00', 'map_index': 0, 'try_number': 1, 'start_date': datetime.datetime(
2025, 11, 2, 23, 4, 2, 810671, tzinfo=Timezone('UTC')), 'end_date': datetime.datetime(2025, 11, 2, 23, 4, 5, 93361, tzinfo=Timezone('UTC')), 'duration': 2.28269, 'state': 'success', 'max_tries': 0, 'host
name': 'airflow-worker-0.airflow-worker.airflow.svc.cluster.local', 'unixname': 'airflow', 'job_id': 721182, 'pool': 'default_pool', 'pool_slots': 1, 'queue': 'default', 'priority_weight': 1, 'operator':
'_PythonDecoratedOperator', 'custom_operator_name': None, 'queued_dttm': datetime.datetime(2025, 11, 2, 23, 4, 1, 165684, tzinfo=Timezone('UTC')), 'queued_by_job_id': 720119, 'pid': 68415, 'executor': N
one, 'executor_config': <psycopg2.extensions.Binary object at 0x7fd2dc58b4b0>, 'updated_at': datetime.datetime(2025, 11, 2, 23, 4, 5, 146802, tzinfo=Timezone('UTC')), 'rendered_map_index': None, 'externa
l_executor_id': 'a96e4277-0d0e-4b53-868d-48e72e2dd9bb', 'trigger_id': None, 'trigger_timeout': None, 'next_method': None, 'next_kwargs': 'null', 'task_display_name': 'nosify.notify_successes'}]
(Background on this error at: https://sqlalche.me/e/14/gkpj)</code>
### What you think should happen instead?
The code doesn't add a value for the `id` column, but it can't be `NULL`, either. This only leaves the conclusion that it should auto-increment.
The `task_instance_history` table was added in `airflow-core/src/airflow/migrations/versions/0021_2_10_0_add_task_instance_history.py` like this:
```py
def upgrade():
"""Add task_instance_history table."""
op.create_table(
"task_instance_history",
sa.Column("id", sa.Integer(), primary_key=True, autoincrement=True),
```
In `airflow-core/src/airflow/migrations/versions/0060_3_0_0_add_try_id_to_ti_and_tih.py` the `id` column is dropped during `upgrade()` and reinstated in `downgrade()` as follows:
```py
def downgrade():
"""Unapply Add try_id to TI and TIH."""
dialect_name = op.get_bind().dialect.name
with op.batch_alter_table("task_instance_history", schema=None) as batch_op:
...
batch_op.add_column(sa.Column("id", sa.INTEGER, nullable=True))
...
...
with op.batch_alter_table("task_instance_history", schema=None) as batch_op:
batch_op.alter_column("id", nullable=False, existing_type=sa.INTEGER)
...
```
So we can see it loses the auto-increment property and is no longer the Pkey.
### How to reproduce
Upgrade the DB to revision `7645189f3479`, then downgrade to `e00344393f31`.
### Operating System
Host: CentOS 7, worker: Debian 12
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downgrading DB leaves instance unable to clear tasks [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed issue #57747: Downgrading DB leaves instance unable to clear tasks
URL: https://github.com/apache/airflow/issues/57747
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downgrading DB leaves instance unable to clear tasks [airflow]
Posted by "karenbraganz (via GitHub)" <gi...@apache.org>.
karenbraganz commented on issue #57747:
URL: https://github.com/apache/airflow/issues/57747#issuecomment-3487997542
@Chais I have assigned this to you since you indicated that you are willing to submit a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downgrading DB leaves instance unable to clear tasks [airflow]
Posted by "karenbraganz (via GitHub)" <gi...@apache.org>.
karenbraganz commented on issue #57747:
URL: https://github.com/apache/airflow/issues/57747#issuecomment-3488024125
@Chais feel free to unassign yourself if you want someone else to take over!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downgrading DB leaves instance unable to clear tasks [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #57747:
URL: https://github.com/apache/airflow/issues/57747#issuecomment-3986428131
@Chais We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downgrading DB leaves instance unable to clear tasks [airflow]
Posted by "karenbraganz (via GitHub)" <gi...@apache.org>.
karenbraganz commented on issue #57747:
URL: https://github.com/apache/airflow/issues/57747#issuecomment-3534432733
@Chais if you are on the Airflow OSS Slack, you could ask someone to review your PR on the #contributors channel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downgrading DB leaves instance unable to clear tasks [airflow]
Posted by "Chais (via GitHub)" <gi...@apache.org>.
Chais commented on issue #57747:
URL: https://github.com/apache/airflow/issues/57747#issuecomment-3526661584
@karenbraganz Could you take a look, or poke someone who can?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downgrading DB leaves instance unable to clear tasks [airflow]
Posted by "Chais (via GitHub)" <gi...@apache.org>.
Chais commented on issue #57747:
URL: https://github.com/apache/airflow/issues/57747#issuecomment-3488008067
If I can figure out how, sure 😄
The migrations seem to be in quite a state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Downgrading DB leaves instance unable to clear tasks [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #57747:
URL: https://github.com/apache/airflow/issues/57747#issuecomment-5942897410
This was fixed by #61976, released in 3.2.0: downgrading past that migration now restores the task_instance_history id sequence and primary key. If you still see this on the latest release, comment here and we'll reopen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org