You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/07 11:39:13 UTC
[GitHub] [airflow] potiuk opened a new pull request #19453: Improve message and documentation around moved data
potiuk opened a new pull request #19453:
URL: https://github.com/apache/airflow/pull/19453
In Airflow 2.2.2 we introduced a fix in #18953 where the corrupted
data was moved to a separate table. However some of our users
(rightly) might not have the context. We've never had anything
like that before, so the users who treat Airflow DB as
black-boxes might get confused on what the error means and what
they should do in this case.
You can see it in #19440 converted into discussion #19444 and #19421
indicate that the message is a bit unclear for users. This PR attempts to
improve that it adds `upgrading` section to our documentation and have the
message link to it so that rather than asking questions in the issues,
users can find context and answers what they should do in our docs.
It also guides the users who treat Airflow DB as "black-box" on how they
can use their tools and airflow db shell to fix the problem.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#discussion_r744302065
##########
File path: airflow/www/templates/airflow/dags.html
##########
@@ -57,7 +57,8 @@
Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
metadatabase, and has moved them to <code>{{ moved_table_name }}</code> during the database migration
to upgrade. Please inspect the moved data to decide whether you need to keep them, and manually drop
- the <code>{{ moved_table_name }}</code> table to dismiss this warning.
+ the <code>{{ moved_table_name }}</code> table to dismiss this warning.<p/>
Review comment:
Intended actually - I added `<p>` so that the link is below the message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#discussion_r744302065
##########
File path: airflow/www/templates/airflow/dags.html
##########
@@ -57,7 +57,8 @@
Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
metadatabase, and has moved them to <code>{{ moved_table_name }}</code> during the database migration
to upgrade. Please inspect the moved data to decide whether you need to keep them, and manually drop
- the <code>{{ moved_table_name }}</code> table to dismiss this warning.
+ the <code>{{ moved_table_name }}</code> table to dismiss this warning.<p/>
Review comment:
Intended actually - I added <p> so that the link is below the message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#discussion_r745084105
##########
File path: airflow/www/templates/airflow/dags.html
##########
@@ -57,7 +57,8 @@
Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
metadatabase, and has moved them to <code>{{ moved_table_name }}</code> during the database migration
to upgrade. Please inspect the moved data to decide whether you need to keep them, and manually drop
- the <code>{{ moved_table_name }}</code> table to dismiss this warning.
+ the <code>{{ moved_table_name }}</code> table to dismiss this warning.<p/>
+ Read more about it in <a href={{ get_docs_url("installing/upgrading.html") }}><b>Upgrading</b></a>
Review comment:
https://stackoverflow.com/a/1261157/5691525 -- suggests some alternatives instead of `<p/>`
Maybe @bbovenzi , @ashb or @ryanahamilton can help here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lukasnalezenec commented on pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
lukasnalezenec commented on pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#issuecomment-964025620
Hi, can we add some simple logging message to code ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19453:
URL: https://github.com/apache/airflow/pull/19453
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#discussion_r745977846
##########
File path: docs/apache-airflow/installation/upgrading.rst
##########
@@ -0,0 +1,77 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Upgrading Airflow to a newer version
+------------------------------------
+
+When you are upgrading Airflow you should always upgrade the metadata DB as part of the upgrade
+process. This might happen automatically in some cases.
Review comment:
I like the idea of adding more sections. I refactored it quite a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#discussion_r746797884
##########
File path: docs/apache-airflow/installation/upgrading.rst
##########
@@ -0,0 +1,94 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Upgrading Airflow to a newer version
+------------------------------------
+
+Why you need to upgrade
+=======================
+
+Newer Airflow versions can contain Database migrations so it is recommended that you run
+``airflow db upgrade`` to Upgrade your Database with the schema changes in the Airflow version
+you are upgrading to.
+
+When you need to upgrade
+========================
+
+If you have a custom deployment based on virtualenv or Docker Containers, you usually need to run
+the DB upgrade manually as part of the upgrade process.
+
+In some cases the upgrade happens automatically - it depends if in your deployment, the upgrade is
+built-in as post-install action. For example when you are using :doc:`helm-chart:index` with
+post-upgrade hooks enabled, the database upgrade happens automatically right after the new software
+is installed. Similarly all Airflow-As-A-Service solutions perform the upgrade automatically for you,
+when you choose to upgrade airflow via their UI
Review comment:
```suggestion
when you choose to upgrade airflow via their UI.
```
##########
File path: airflow/www/templates/airflow/dags.html
##########
@@ -57,7 +57,8 @@
Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
metadatabase, and has moved them to <code>{{ moved_table_name }}</code> during the database migration
to upgrade. Please inspect the moved data to decide whether you need to keep them, and manually drop
- the <code>{{ moved_table_name }}</code> table to dismiss this warning.
+ the <code>{{ moved_table_name }}</code> table to dismiss this warning. Read more about it
+ in <a href={{ get_docs_url("installing/upgrading.html") }}><b>Upgrading</b></a>
Review comment:
```suggestion
in <a href={{ get_docs_url("installing/upgrading.html") }}><b>Upgrading</b></a>.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#issuecomment-965550115
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#discussion_r744303240
##########
File path: airflow/www/templates/airflow/dags.html
##########
@@ -57,7 +57,8 @@
Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
metadatabase, and has moved them to <code>{{ moved_table_name }}</code> during the database migration
to upgrade. Please inspect the moved data to decide whether you need to keep them, and manually drop
- the <code>{{ moved_table_name }}</code> table to dismiss this warning.
+ the <code>{{ moved_table_name }}</code> table to dismiss this warning.<p/>
Review comment:
Not really - more to have the link very close on the left side rather than in a very unspecified place depending on the screen size:
Other suggestions how it can be done better if not via paragraph?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#issuecomment-964505596
Thanks for the commends @jedcunningham @uranusjr @kaxil - I think it's much better now :)
Proposal: if it won't make it in 2.2.2 (I doubt now) - then we can cherry-pick that one to v2-2-test and build docs using it so that at least the docs are published with 2.2.2 release :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#discussion_r744303240
##########
File path: airflow/www/templates/airflow/dags.html
##########
@@ -57,7 +57,8 @@
Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
metadatabase, and has moved them to <code>{{ moved_table_name }}</code> during the database migration
to upgrade. Please inspect the moved data to decide whether you need to keep them, and manually drop
- the <code>{{ moved_table_name }}</code> table to dismiss this warning.
+ the <code>{{ moved_table_name }}</code> table to dismiss this warning.<p/>
Review comment:
Not really - more to have the link very close on the right side rather than in a very unspecified place depending on the screen size:
Other suggestions how it can be done better if not via paragraph?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#issuecomment-963410671
Should we merge that one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#issuecomment-964501613
> Hi, can we add some simple logging message to code ?
`airflow db upgrade` does not have logging. It will print the output to stdout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#discussion_r745085944
##########
File path: docs/apache-airflow/installation/upgrading.rst
##########
@@ -0,0 +1,77 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Upgrading Airflow to a newer version
+------------------------------------
+
+When you are upgrading Airflow you should always upgrade the metadata DB as part of the upgrade
+process. This might happen automatically in some cases.
Review comment:
```suggestion
Newer Airflow versions can contain Database migrations so it is recommended that you run ``airflow db upgrade`` to Upgrade your Database with the schema changes in the Airflow version you are upgrading to.
```
Or have a separate Why you to need upgrade and have that para
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#discussion_r745084105
##########
File path: airflow/www/templates/airflow/dags.html
##########
@@ -57,7 +57,8 @@
Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
metadatabase, and has moved them to <code>{{ moved_table_name }}</code> during the database migration
to upgrade. Please inspect the moved data to decide whether you need to keep them, and manually drop
- the <code>{{ moved_table_name }}</code> table to dismiss this warning.
+ the <code>{{ moved_table_name }}</code> table to dismiss this warning.<p/>
+ Read more about it in <a href={{ get_docs_url("installing/upgrading.html") }}><b>Upgrading</b></a>
Review comment:
https://stackoverflow.com/a/1261157/5691525 -- suggests some alternatives instead of `<p/>`
Maybe @bbovenzi or @ryanahamilton can help here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#discussion_r744302639
##########
File path: airflow/www/templates/airflow/dags.html
##########
@@ -57,7 +57,8 @@
Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
metadatabase, and has moved them to <code>{{ moved_table_name }}</code> during the database migration
to upgrade. Please inspect the moved data to decide whether you need to keep them, and manually drop
- the <code>{{ moved_table_name }}</code> table to dismiss this warning.
+ the <code>{{ moved_table_name }}</code> table to dismiss this warning.<p/>
Review comment:
If the intention is to split the two parts, I'm not sure this is a recommended way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#discussion_r744301529
##########
File path: airflow/www/templates/airflow/dags.html
##########
@@ -57,7 +57,8 @@
Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
metadatabase, and has moved them to <code>{{ moved_table_name }}</code> during the database migration
to upgrade. Please inspect the moved data to decide whether you need to keep them, and manually drop
- the <code>{{ moved_table_name }}</code> table to dismiss this warning.
+ the <code>{{ moved_table_name }}</code> table to dismiss this warning.<p/>
Review comment:
This looks unintended?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#discussion_r745971927
##########
File path: airflow/www/templates/airflow/dags.html
##########
@@ -57,7 +57,8 @@
Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
metadatabase, and has moved them to <code>{{ moved_table_name }}</code> during the database migration
to upgrade. Please inspect the moved data to decide whether you need to keep them, and manually drop
- the <code>{{ moved_table_name }}</code> table to dismiss this warning.
+ the <code>{{ moved_table_name }}</code> table to dismiss this warning.<p/>
+ Read more about it in <a href={{ get_docs_url("installing/upgrading.html") }}><b>Upgrading</b></a>
Review comment:
In the end it's not that important I removed the paragraph
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#discussion_r745078133
##########
File path: airflow/www/templates/airflow/dags.html
##########
@@ -57,7 +57,8 @@
Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
Review comment:
```suggestion
<p>Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
```
If we are going to close a paragraph, we should start one too.
##########
File path: docs/apache-airflow/installation/upgrading.rst
##########
@@ -0,0 +1,77 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Upgrading Airflow to a newer version
+------------------------------------
+
+When you are upgrading Airflow you should always upgrade the metadata DB as part of the upgrade
+process. This might happen automatically in some cases.
+For example when you are using :doc:`helm-chart:index` with post-upgrade hooks enabled, the database
+upgrade happens automatically after upgrade is completed. Similarly all Airflow-As-A-Service solutions
+run the upgrade automatically for you.
+
+However if you have your custom deployment based on virtualenv or Docker Containers, you need to run
Review comment:
```suggestion
However if you have a custom deployment based on virtualenv or Docker Containers, you need to run
```
##########
File path: docs/apache-airflow/installation/upgrading.rst
##########
@@ -0,0 +1,77 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Upgrading Airflow to a newer version
+------------------------------------
+
+When you are upgrading Airflow you should always upgrade the metadata DB as part of the upgrade
+process. This might happen automatically in some cases.
+For example when you are using :doc:`helm-chart:index` with post-upgrade hooks enabled, the database
+upgrade happens automatically after upgrade is completed. Similarly all Airflow-As-A-Service solutions
+run the upgrade automatically for you.
Review comment:
```suggestion
upgrade happens automatically after the helm upgrade is completed. Similarly all Airflow-As-A-Service
solutions run the upgrade automatically for you.
```
##########
File path: docs/apache-airflow/installation/upgrading.rst
##########
@@ -0,0 +1,77 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Upgrading Airflow to a newer version
+------------------------------------
+
+When you are upgrading Airflow you should always upgrade the metadata DB as part of the upgrade
+process. This might happen automatically in some cases.
+For example when you are using :doc:`helm-chart:index` with post-upgrade hooks enabled, the database
+upgrade happens automatically after upgrade is completed. Similarly all Airflow-As-A-Service solutions
+run the upgrade automatically for you.
+
+However if you have your custom deployment based on virtualenv or Docker Containers, you need to run
+the DB upgrade manually as part of the upgrade process.
+
+In order to manually upgrade the database you should run ``airflow db upgrade`` command in your environment
+(either in your virtual environment or in your containers that gives you access to Airflow ``CLI``:
+:doc:`/usage-cli` and the database.
+
+Depending on the size of your database and the actual migration it might take quite some time to migrate it,
+so if you have long history and big database, it is recommended to make a copy of the database first and
+perform test migration to assess how long the migration will take. Typically "Major" upgrades might take
+longer as adding new features require sometimes restructuring of the database.
+
+Post-upgrade warning messages
+-----------------------------
+
+Typically you just need to successfully run ``airflow db upgrade`` command and this is all. However in
+some cases, the migration might find some old, stale and probably wrong data in your database and move it
+aside to a separate table. In this case you might get warning in your webserver UI about the data found.
+
+Typical message that you might see:
+
+ Airflow found incompatible data in the <original table> table in the
+ metadatabase, and has moved them to <new table> during the database migration to upgrade.
+ Please inspect the moved data to decide whether you need to keep them,
+ and manually drop the <new table> table to dismiss this warning.
+
+When you see such message, it means that some of your data was corrupted and you should inspect it
+whether you would like to keep or move some of that data. Most likely the data was really corrupted
+and left-over from some bugs and can be safely deleted - because this data would not be anyhow visible
+and useful in Airflow. However if you have particular need for auditing or historical reasons you might
+choose to store it somewhere. Unless you have specific reasons to keep the data most likely deleting it
+is your best option.
+
+There are various ways you can inspect and delete the data - if you have direct access to the
+database using your own tools (often graphical tools showing the database objects), you can drop such
+table or rename it or moving to another database using those tools. If you don't have such tools you
Review comment:
```suggestion
table or rename it or move it to another database using those tools. If you don't have such tools you
```
##########
File path: docs/apache-airflow/installation/upgrading.rst
##########
@@ -0,0 +1,77 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Upgrading Airflow to a newer version
+------------------------------------
+
+When you are upgrading Airflow you should always upgrade the metadata DB as part of the upgrade
+process. This might happen automatically in some cases.
+For example when you are using :doc:`helm-chart:index` with post-upgrade hooks enabled, the database
+upgrade happens automatically after upgrade is completed. Similarly all Airflow-As-A-Service solutions
+run the upgrade automatically for you.
+
+However if you have your custom deployment based on virtualenv or Docker Containers, you need to run
+the DB upgrade manually as part of the upgrade process.
+
+In order to manually upgrade the database you should run ``airflow db upgrade`` command in your environment
Review comment:
```suggestion
In order to manually upgrade the database you should run the ``airflow db upgrade`` command in your environment
```
##########
File path: docs/apache-airflow/installation/upgrading.rst
##########
@@ -0,0 +1,77 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Upgrading Airflow to a newer version
+------------------------------------
+
+When you are upgrading Airflow you should always upgrade the metadata DB as part of the upgrade
+process. This might happen automatically in some cases.
+For example when you are using :doc:`helm-chart:index` with post-upgrade hooks enabled, the database
+upgrade happens automatically after upgrade is completed. Similarly all Airflow-As-A-Service solutions
+run the upgrade automatically for you.
+
+However if you have your custom deployment based on virtualenv or Docker Containers, you need to run
+the DB upgrade manually as part of the upgrade process.
+
+In order to manually upgrade the database you should run ``airflow db upgrade`` command in your environment
+(either in your virtual environment or in your containers that gives you access to Airflow ``CLI``:
+:doc:`/usage-cli` and the database.
+
+Depending on the size of your database and the actual migration it might take quite some time to migrate it,
+so if you have long history and big database, it is recommended to make a copy of the database first and
+perform test migration to assess how long the migration will take. Typically "Major" upgrades might take
+longer as adding new features require sometimes restructuring of the database.
+
+Post-upgrade warning messages
+-----------------------------
+
+Typically you just need to successfully run ``airflow db upgrade`` command and this is all. However in
+some cases, the migration might find some old, stale and probably wrong data in your database and move it
+aside to a separate table. In this case you might get warning in your webserver UI about the data found.
+
+Typical message that you might see:
+
+ Airflow found incompatible data in the <original table> table in the
+ metadatabase, and has moved them to <new table> during the database migration to upgrade.
+ Please inspect the moved data to decide whether you need to keep them,
+ and manually drop the <new table> table to dismiss this warning.
+
+When you see such message, it means that some of your data was corrupted and you should inspect it
+whether you would like to keep or move some of that data. Most likely the data was really corrupted
+and left-over from some bugs and can be safely deleted - because this data would not be anyhow visible
+and useful in Airflow. However if you have particular need for auditing or historical reasons you might
+choose to store it somewhere. Unless you have specific reasons to keep the data most likely deleting it
+is your best option.
+
+There are various ways you can inspect and delete the data - if you have direct access to the
+database using your own tools (often graphical tools showing the database objects), you can drop such
+table or rename it or moving to another database using those tools. If you don't have such tools you
+can use ``airflow db shell`` command - this will drop you in the db shell tool for your database and you
+will be able to both inspect and delete the table (Replace ``<table>`` in the examples with the actual
+table name as printed in the warning message.
Review comment:
```suggestion
table name as printed in the warning message).
```
##########
File path: docs/apache-airflow/installation/upgrading.rst
##########
@@ -0,0 +1,77 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Upgrading Airflow to a newer version
+------------------------------------
+
+When you are upgrading Airflow you should always upgrade the metadata DB as part of the upgrade
+process. This might happen automatically in some cases.
+For example when you are using :doc:`helm-chart:index` with post-upgrade hooks enabled, the database
+upgrade happens automatically after upgrade is completed. Similarly all Airflow-As-A-Service solutions
+run the upgrade automatically for you.
+
+However if you have your custom deployment based on virtualenv or Docker Containers, you need to run
+the DB upgrade manually as part of the upgrade process.
+
+In order to manually upgrade the database you should run ``airflow db upgrade`` command in your environment
+(either in your virtual environment or in your containers that gives you access to Airflow ``CLI``:
+:doc:`/usage-cli` and the database.
+
+Depending on the size of your database and the actual migration it might take quite some time to migrate it,
+so if you have long history and big database, it is recommended to make a copy of the database first and
+perform test migration to assess how long the migration will take. Typically "Major" upgrades might take
+longer as adding new features require sometimes restructuring of the database.
+
+Post-upgrade warning messages
+-----------------------------
+
+Typically you just need to successfully run ``airflow db upgrade`` command and this is all. However in
+some cases, the migration might find some old, stale and probably wrong data in your database and move it
+aside to a separate table. In this case you might get warning in your webserver UI about the data found.
+
+Typical message that you might see:
+
+ Airflow found incompatible data in the <original table> table in the
+ metadatabase, and has moved them to <new table> during the database migration to upgrade.
+ Please inspect the moved data to decide whether you need to keep them,
+ and manually drop the <new table> table to dismiss this warning.
+
+When you see such message, it means that some of your data was corrupted and you should inspect it
+whether you would like to keep or move some of that data. Most likely the data was really corrupted
+and left-over from some bugs and can be safely deleted - because this data would not be anyhow visible
+and useful in Airflow. However if you have particular need for auditing or historical reasons you might
+choose to store it somewhere. Unless you have specific reasons to keep the data most likely deleting it
+is your best option.
+
+There are various ways you can inspect and delete the data - if you have direct access to the
+database using your own tools (often graphical tools showing the database objects), you can drop such
+table or rename it or moving to another database using those tools. If you don't have such tools you
+can use ``airflow db shell`` command - this will drop you in the db shell tool for your database and you
Review comment:
```suggestion
can use the ``airflow db shell`` command - this will drop you in the db shell tool for your database and you
```
##########
File path: docs/apache-airflow/installation/upgrading.rst
##########
@@ -0,0 +1,77 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Upgrading Airflow to a newer version
+------------------------------------
+
+When you are upgrading Airflow you should always upgrade the metadata DB as part of the upgrade
+process. This might happen automatically in some cases.
+For example when you are using :doc:`helm-chart:index` with post-upgrade hooks enabled, the database
+upgrade happens automatically after upgrade is completed. Similarly all Airflow-As-A-Service solutions
+run the upgrade automatically for you.
+
+However if you have your custom deployment based on virtualenv or Docker Containers, you need to run
+the DB upgrade manually as part of the upgrade process.
+
+In order to manually upgrade the database you should run ``airflow db upgrade`` command in your environment
+(either in your virtual environment or in your containers that gives you access to Airflow ``CLI``:
+:doc:`/usage-cli` and the database.
+
+Depending on the size of your database and the actual migration it might take quite some time to migrate it,
+so if you have long history and big database, it is recommended to make a copy of the database first and
+perform test migration to assess how long the migration will take. Typically "Major" upgrades might take
+longer as adding new features require sometimes restructuring of the database.
+
+Post-upgrade warning messages
+-----------------------------
+
+Typically you just need to successfully run ``airflow db upgrade`` command and this is all. However in
+some cases, the migration might find some old, stale and probably wrong data in your database and move it
Review comment:
```suggestion
some cases, the migration might find some old, stale and probably wrong data in your database and moves it
```
##########
File path: docs/apache-airflow/installation/upgrading.rst
##########
@@ -0,0 +1,77 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Upgrading Airflow to a newer version
+------------------------------------
+
+When you are upgrading Airflow you should always upgrade the metadata DB as part of the upgrade
+process. This might happen automatically in some cases.
+For example when you are using :doc:`helm-chart:index` with post-upgrade hooks enabled, the database
+upgrade happens automatically after upgrade is completed. Similarly all Airflow-As-A-Service solutions
+run the upgrade automatically for you.
+
+However if you have your custom deployment based on virtualenv or Docker Containers, you need to run
+the DB upgrade manually as part of the upgrade process.
+
+In order to manually upgrade the database you should run ``airflow db upgrade`` command in your environment
+(either in your virtual environment or in your containers that gives you access to Airflow ``CLI``:
+:doc:`/usage-cli` and the database.
+
+Depending on the size of your database and the actual migration it might take quite some time to migrate it,
+so if you have long history and big database, it is recommended to make a copy of the database first and
+perform test migration to assess how long the migration will take. Typically "Major" upgrades might take
+longer as adding new features require sometimes restructuring of the database.
+
+Post-upgrade warning messages
+-----------------------------
+
+Typically you just need to successfully run ``airflow db upgrade`` command and this is all. However in
Review comment:
```suggestion
Typically you just need to successfully run the ``airflow db upgrade`` command and this is all. However in
```
##########
File path: docs/apache-airflow/installation/upgrading.rst
##########
@@ -0,0 +1,77 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Upgrading Airflow to a newer version
+------------------------------------
+
+When you are upgrading Airflow you should always upgrade the metadata DB as part of the upgrade
+process. This might happen automatically in some cases.
+For example when you are using :doc:`helm-chart:index` with post-upgrade hooks enabled, the database
+upgrade happens automatically after upgrade is completed. Similarly all Airflow-As-A-Service solutions
+run the upgrade automatically for you.
+
+However if you have your custom deployment based on virtualenv or Docker Containers, you need to run
+the DB upgrade manually as part of the upgrade process.
+
+In order to manually upgrade the database you should run ``airflow db upgrade`` command in your environment
+(either in your virtual environment or in your containers that gives you access to Airflow ``CLI``:
+:doc:`/usage-cli` and the database.
+
+Depending on the size of your database and the actual migration it might take quite some time to migrate it,
+so if you have long history and big database, it is recommended to make a copy of the database first and
+perform test migration to assess how long the migration will take. Typically "Major" upgrades might take
+longer as adding new features require sometimes restructuring of the database.
+
+Post-upgrade warning messages
+-----------------------------
+
+Typically you just need to successfully run ``airflow db upgrade`` command and this is all. However in
+some cases, the migration might find some old, stale and probably wrong data in your database and move it
+aside to a separate table. In this case you might get warning in your webserver UI about the data found.
+
+Typical message that you might see:
+
+ Airflow found incompatible data in the <original table> table in the
+ metadatabase, and has moved them to <new table> during the database migration to upgrade.
+ Please inspect the moved data to decide whether you need to keep them,
+ and manually drop the <new table> table to dismiss this warning.
+
+When you see such message, it means that some of your data was corrupted and you should inspect it
+whether you would like to keep or move some of that data. Most likely the data was really corrupted
Review comment:
```suggestion
When you see such message, it means that some of your data was corrupted and you should inspect it to determine
whether you would like to keep or delete that data. Most likely the data was really corrupted
```
##########
File path: docs/apache-airflow/installation/upgrading.rst
##########
@@ -0,0 +1,77 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Upgrading Airflow to a newer version
+------------------------------------
+
+When you are upgrading Airflow you should always upgrade the metadata DB as part of the upgrade
+process. This might happen automatically in some cases.
+For example when you are using :doc:`helm-chart:index` with post-upgrade hooks enabled, the database
+upgrade happens automatically after upgrade is completed. Similarly all Airflow-As-A-Service solutions
+run the upgrade automatically for you.
+
+However if you have your custom deployment based on virtualenv or Docker Containers, you need to run
+the DB upgrade manually as part of the upgrade process.
+
+In order to manually upgrade the database you should run ``airflow db upgrade`` command in your environment
+(either in your virtual environment or in your containers that gives you access to Airflow ``CLI``:
+:doc:`/usage-cli` and the database.
+
+Depending on the size of your database and the actual migration it might take quite some time to migrate it,
+so if you have long history and big database, it is recommended to make a copy of the database first and
+perform test migration to assess how long the migration will take. Typically "Major" upgrades might take
Review comment:
```suggestion
perform a test migration to assess how long the migration will take. Typically "Major" upgrades might take
```
##########
File path: airflow/www/templates/airflow/dags.html
##########
@@ -57,7 +57,8 @@
Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
metadatabase, and has moved them to <code>{{ moved_table_name }}</code> during the database migration
to upgrade. Please inspect the moved data to decide whether you need to keep them, and manually drop
- the <code>{{ moved_table_name }}</code> table to dismiss this warning.
+ the <code>{{ moved_table_name }}</code> table to dismiss this warning.<p/>
+ Read more about it in <a href={{ get_docs_url("installing/upgrading.html") }}><b>Upgrading</b></a>
Review comment:
```suggestion
<p>Read more about it in <a href={{ get_docs_url("installing/upgrading.html") }}><b>Upgrading</b></a></p>
```
And then this should be a paragraph as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #19453: Improve message and documentation around moved data
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #19453:
URL: https://github.com/apache/airflow/pull/19453#discussion_r745092601
##########
File path: airflow/www/templates/airflow/dags.html
##########
@@ -57,7 +57,8 @@
Airflow found incompatible data in the <code>{{ original_table_name }}</code> table in the
metadatabase, and has moved them to <code>{{ moved_table_name }}</code> during the database migration
to upgrade. Please inspect the moved data to decide whether you need to keep them, and manually drop
- the <code>{{ moved_table_name }}</code> table to dismiss this warning.
+ the <code>{{ moved_table_name }}</code> table to dismiss this warning.<p/>
Review comment:
```suggestion
the <code>{{ moved_table_name }}</code> table to dismiss this warning.</p>
```
`p` shouldn't be a self closing tag. We should either wrap both sections in `p`, or use ` br` here. I don't have a strong preference which.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org