[GitHub] [airflow] kaxil commented on a change in pull request #12872: Added documentation for Airflow Upgrade Check

[GitBox <gi...@apache.org>]

kaxil commented on a change in pull request #12872:
URL: https://github.com/apache/airflow/pull/12872#discussion_r537945229



##########
File path: docs/apache-airflow/upgrade-check.rst
##########
@@ -0,0 +1,145 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+
+Upgrade Check Script
+--------------------
+
+.. contents:: :local:
+
+Getting the Airflow Upgrade Check Package
+'''''''''''''''''''''''''''''''''''''''''
+
+Apache Airflow is published as ``apache-airflow`` package in PyPI. The Upgrade Check Script is part of a
+separate Python Package, since it is separate from the core Apache Airflow package and is only needed for
+a period of time and specifically only for upgrading from Airflow 1.10 releases to Airflow 2.0.
+
+While there has been a lot of work put into making this upgrade as easy as possible, there are some
+changes which are compatible between Airflow 1.10 and Airflow 2.0. In order to make this as simple to
+navigate, we recommend that people first upgrade to the latest release in the 1.10 series (at the
+time of writing: 1.10.14) and then to download this package and run the script as detailed below.
+
+
+.. note::
+
+   On November 2020, new version of PIP (20.3) has been released with a new, 2020 resolver. This resolver
+   does not yet work with Apache Airflow and might leads to errors in installation - depends on your choice
+   of extras. In order to install Airflow you need to either downgrade pip to version 20.2.4
+   ``pip upgrade --pip==20.2.4`` or, in case you use Pip 20.3, you need to add option
+   ``--use-deprecated legacy-resolver`` to your pip install command.
+
+.. code-block:: bash
+
+    pip install apache-airflow-upgrade-check
+
+This will install the latest version of the Airflow Upgrade check package.
+
+
+Running the Airflow Upgrade Check Package
+'''''''''''''''''''''''''''''''''''''''''
+
+.. code-block:: bash
+
+    airflow upgrade_check
+
+This will print out to the screen a number of actions that should be taken before upgrading the Airflow
+release to 2.0.0 or higher.
+
+The exit code of the command will be 0 (success) if no problems are found in running the command, or 1 if
+problems were encountered in running the check.
+
+A sample output as a result of a successful run of the upgrade check is shown below.
+
+.. code-block:: bash
+
+    ============================================== STATUS ============================================
+
+    Check for latest versions of apache-airflow and checker..................................SUCCESS
+    Remove airflow.AirflowMacroPlugin class..................................................SUCCESS
+    Chain between DAG and operator not allowed...............................................SUCCESS
+    Connection.conn_id is not unique.........................................................SUCCESS
+    Connection.conn_type is not nullable.....................................................SUCCESS
+    Fernet is enabled by default.............................................................FAIL
+    GCP service account key deprecation......................................................SUCCESS
+    Changes in import paths of hooks, operators, sensors and others..........................FAIL
+    Users must delete deprecated configs for KubernetesExecutor..............................FAIL
+    Legacy UI is deprecated by default.......................................................SUCCESS
+    Logging configuration has been moved to new section......................................FAIL
+    Removal of Mesos Executor................................................................SUCCESS
+    Users must set a kubernetes.pod_template_file value......................................FAIL
+    SendGrid email uses old airflow.contrib module...........................................SUCCESS
+    Changes in import path of remote task handlers...........................................SUCCESS
+    Jinja Template Variables cannot be undefined.............................................FAIL
+    Found 7 problems.
+
+    ========================================== RECOMMENDATIONS ========================================
+
+    Fernet is enabled by default
+    ----------------------------
+    The fernet mechanism is enabled by default to increase the security of the default installation.
+
+    Problems:
+
+    1.  fernet_key in airflow.cfg must be explicitly set empty as fernet mechanism is enabledby default. This means that the apache-airflow[crypto] extra-packages are always installed.However, this requires that your operating system has libffi-dev installed.
+
+    Changes in import paths of hooks, operators, sensors and others
+    ---------------------------------------------------------------
+    Many hooks, operators and other classes has been renamed and moved. Those changes were part of unifying names and imports paths as described in AIP-21.
+    The contrib folder has been replaced by providers directory and packages:
+    https://github.com/apache/airflow#backport-packages
+
+    Problems:
+
+    1.  Using airflow.operators.python_operator.PythonOperator will be replaced by airflow.operators.python.PythonOperator. Affected file:
+
+
+The following sections describe what is being done and how to apply the recommendations shown above.
+Please note that the above results shown are only a partial set, where only the first
+two of the seven problems identified are shown in the section above. In reality,
+all the problems are shown on the screen.
+
+
+Understanding what is being checked
+'''''''''''''''''''''''''''''''''''
+
+The Upgrade Check checks the configuration data from airflow.cfg, the meta data from the Airflow
+database, as well as the DAGs which have been set up in the current Airflow environment.
+
+Using the above results as an example, there are two specific problems which have
+been identified.
+
+The first problem is identified in the configuration file airflow.cfg where the current configuration
+option for the fernet_key is no longer acceptable and needs to be changed.
+
+The second problem was identified in one of the DAGs. In this case, this import
+statement for the PythonOperator needs to be changed, since the location is different
+in Airflow 2.0. We will discuss how in the next section.

Review comment:
       ```suggestion
   Understanding what is being checked
   '''''''''''''''''''''''''''''''''''
   
   The Upgrade Check checks the configuration data from airflow.cfg, the metadata from the Airflow
   database, as well as the DAGs which have been set up in the current Airflow environment.
   
   Using the above results as an example, there are two specific problems which have
   been identified.
   
   The first problem is identified in the configuration file airflow.cfg where the current configuration
   option for the fernet_key is no longer acceptable and needs to be changed.
   
   The second problem was identified in one of the DAGs. In this case, this import
   statement for the PythonOperator needs to be changed, since the location is different
   in Airflow 2.0. We will discuss how in the next section.
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org