You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/13 16:52:22 UTC
[GitHub] [airflow] uranusjr opened a new pull request #18953: Try to move "dangling" rows in upgradedb
uranusjr opened a new pull request #18953:
URL: https://github.com/apache/airflow/pull/18953
Fix #18894.
Instead of failing loudly for invalid records (which happens way too often), this attempts to move those offending data to another table and carry on with the migration if possible. This table for dangling data are copied with `CREATE TABLE ... AS SELECT ...` and could miss some indexing and stuff, but this is only meant for temporary storage, so this is probably not a big deal. If copying went well, the dangling data are automatically deleted so we can carry on with migration.
This also _removed_ the upgrade check on TaskFail and _added_ TaskReschedule. This is because TaskFail is not actually being migrated in 2.2, while TaskReschedule is, and we concluded this is likely a typo during implementation and not an intentional choice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #18953: Try to move "dangling" rows in upgradedb
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #18953:
URL: https://github.com/apache/airflow/pull/18953#issuecomment-943375300
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #18953: Try to move "dangling" rows in upgradedb
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #18953:
URL: https://github.com/apache/airflow/pull/18953#discussion_r729094722
##########
File path: airflow/www/templates/airflow/dags.html
##########
@@ -51,6 +51,14 @@
{% for m in dashboard_alerts %}
{{ message(m.message, m.category) }}
{% endfor %}
+ {% for original_table_name, moved_table_name in migration_moved_data_alerts %}
+ {% call message(category='error', dismissable=false) %}
+ Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
+ metadatabase, and have moved them to <code>{{ moved_table_name }}</code> during migration to
+ perform version upgrade. Please inspect the moved data to decide whether you need to keep them,
Review comment:
I wasn't so sure about my own wording, this is nice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #18953: Try to move "dangling" rows in upgradedb
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #18953:
URL: https://github.com/apache/airflow/pull/18953#issuecomment-943366260
I just added some code to show the alert in the web UI. This should be ready.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #18953: Try to move "dangling" rows in upgradedb
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #18953:
URL: https://github.com/apache/airflow/pull/18953#discussion_r729076205
##########
File path: airflow/www/templates/airflow/dags.html
##########
@@ -51,6 +51,14 @@
{% for m in dashboard_alerts %}
{{ message(m.message, m.category) }}
{% endfor %}
+ {% for original_table_name, moved_table_name in migration_moved_data_alerts %}
+ {% call message(category='error', dismissable=false) %}
+ Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
+ metadatabase, and have moved them to <code>{{ moved_table_name }}</code> during migration to
+ perform version upgrade. Please inspect the moved data to decide whether you need to keep them,
Review comment:
```suggestion
metadatabase, and has moved them to <code>{{ moved_table_name }}</code> during the database
version upgrade. Please inspect the moved data to decide whether you need to keep them,
```
Nit, sounds better to my ear?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #18953: Try to move "dangling" rows in upgradedb
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #18953:
URL: https://github.com/apache/airflow/pull/18953#issuecomment-943466398
Also I somehow forgot to push the fix to the db query count test before :( Waiting for CI to pass...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org