You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/07 10:04:47 UTC
[GitHub] [airflow] robinedwards opened a new issue #18801: Migrations fail to apply when there are null run_id's in the dag_run table
robinedwards opened a new issue #18801:
URL: https://github.com/apache/airflow/issues/18801
### Apache Airflow version
2.2.0b2 (beta snapshot)
### Operating System
Debian GNU/Linux 11 (bullseye)
### Versions of Apache Airflow Providers
```
apache-airflow-providers-amazon @ file:///root/.cache/pypoetry/artifacts/c9/69/16/ffa2eb7a2e6e850a7048eaf66b6c40c990ef7c58149f20d3d3f333a2e9/apache_airflow_providers_amazon-2.2.0-py3-none-any.whl
apache-airflow-providers-celery @ file:///root/.cache/pypoetry/artifacts/6e/1b/2f/f968318a7474e979af4dc53893ecafe8cd11a98a94077a9c3c27304eb7/apache_airflow_providers_celery-2.1.0-py3-none-any.whl
apache-airflow-providers-ftp @ file:///root/.cache/pypoetry/artifacts/8b/9a/dd/79a36c62bc7f37f98d0ea33652570e19272e8a7a2297db13a6785698d1/apache_airflow_providers_ftp-2.0.1-py3-none-any.whl
apache-airflow-providers-http @ file:///root/.cache/pypoetry/artifacts/52/28/81/03a89147daf7daceb55f1218189d1c4af01c33c45849b568769ca6765f/apache_airflow_providers_http-2.0.1-py3-none-any.whl
apache-airflow-providers-imap @ file:///root/.cache/pypoetry/artifacts/1c/5d/c5/269e8a8098e7017a26a2a376eb3020e1a864775b7ff310ed39e1bd503d/apache_airflow_providers_imap-2.0.1-py3-none-any.whl
apache-airflow-providers-postgres @ file:///root/.cache/pypoetry/artifacts/fb/69/ac/e8e25a0f6a4b0daf162c81c9cfdbb164a93bef6bd652c1c00eee6e0815/apache_airflow_providers_postgres-2.3.0-py3-none-any.whl
apache-airflow-providers-redis @ file:///root/.cache/pypoetry/artifacts/cf/2b/56/75563b6058fe45b70f93886dd92541e8349918eeea9d70c703816f2639/apache_airflow_providers_redis-2.0.1-py3-none-any.whl
apache-airflow-providers-sqlite @ file:///root/.cache/pypoetry/artifacts/61/ba/e9/c0b4b7ef2599dbd902b32afc99f2620d8a616b3072122e90f591de4807/apache_airflow_providers_sqlite-2.0.1-py3-none-any.whl
```
### Deployment
Other Docker-based deployment
### Deployment details
AWS ECS + Fargate, RabbitMQ, Celery executor, Postgres 13 backend
### What happened
Hey so we just hit an issue with the 2.2.0b2 migration https://github.com/apache/airflow/blob/63b1c270cb09ff897702dafda28c59559f9e6080/airflow/migrations/versions/7b2661a43ba3_taskinstance_keyed_to_dagrun.py#L1
. We have dag_runs without a run_id (as it is still nullable) on the dag_run table. This caused the FK constraint on TI not to apply.
Noticeably this wasn't caught by https://github.com/apache/airflow/blob/ca93ff4ef05e68369b4fd42fcda3b40f63682d4a/airflow/utils/db.py#L700 before the migrations apply. Is it worth adding an additional check here and making dag_run.run_id NOT NULL?
### What you expected to happen
The data check before applying migrations should have spotted the NULL run_ids on the dag_run table. Going forward perhaps we can make this column NOT NULL?
### How to reproduce
NULL some run_id columns on the dag_run table. As to how this occurs I am not sure.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] robinedwards commented on issue #18801: Migrations fail to apply when there are null run_id's in the dag_run table
Posted by GitBox <gi...@apache.org>.
robinedwards commented on issue #18801:
URL: https://github.com/apache/airflow/issues/18801#issuecomment-937663278
Does the run_id on dag_run genuinely need to be nullable? Feels like having it nullable will potentially create issues down the line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] robinedwards commented on issue #18801: Migrations fail to apply when there are null run_id's in the dag_run table
Posted by GitBox <gi...@apache.org>.
robinedwards commented on issue #18801:
URL: https://github.com/apache/airflow/issues/18801#issuecomment-937663278
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #18801: Migrations fail to apply when there are null run_id's in the dag_run table
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #18801:
URL: https://github.com/apache/airflow/issues/18801#issuecomment-937658418
We should add a upgrade check to catch this then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #18801: Migrations fail to apply when there are null run_id's in the dag_run table
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #18801:
URL: https://github.com/apache/airflow/issues/18801#issuecomment-937688966
No, it shouldn't be nullable.
Do you have any idea how it happened?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] robinedwards commented on issue #18801: Migrations fail to apply when there are null run_id's in the dag_run table
Posted by GitBox <gi...@apache.org>.
robinedwards commented on issue #18801:
URL: https://github.com/apache/airflow/issues/18801#issuecomment-937878374
Unfortunately I don't, there were some fairly recent examples of this occurring (execution date in the last month). We've recently upgraded from 2.0.2 to 2.1.3 as it solved a couple of issues for us and as the 2.2.0 beta solved some more we are rolling out that. We'd already done this successfully on a small integration environment. We will be rolling out our live environment at some point next week and i can have a look then to see if there are any patterns
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed issue #18801: Migrations fail to apply when there are null run_id's in the dag_run table
Posted by GitBox <gi...@apache.org>.
ashb closed issue #18801:
URL: https://github.com/apache/airflow/issues/18801
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #18801: Migrations fail to apply when there are null run_id's in the dag_run table
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #18801:
URL: https://github.com/apache/airflow/issues/18801#issuecomment-938451868
@robinedwards We fixed this, but have taken the decision to _alter_ the existing migration. I think this should be fine for you as the previous migration wouldn't have applied cleanly.
If on the other hand you are using mysql you have a _partly_ applied migration then you'll need to manually edit that database to unwind the migration or fully apply it and updates `alembic_version` table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #18801: Migrations fail to apply when there are null run_id's in the dag_run table
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #18801:
URL: https://github.com/apache/airflow/issues/18801#issuecomment-937688966
No, it shouldn't be nullable.
Do you have any idea how it happened?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #18801: Migrations fail to apply when there are null run_id's in the dag_run table
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #18801:
URL: https://github.com/apache/airflow/issues/18801#issuecomment-937658418
We should add a upgrade check to catch this then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] robinedwards commented on issue #18801: Migrations fail to apply when there are null run_id's in the dag_run table
Posted by GitBox <gi...@apache.org>.
robinedwards commented on issue #18801:
URL: https://github.com/apache/airflow/issues/18801#issuecomment-938460337
Thanks for that Ash much appreciated :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org