You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/16 21:53:53 UTC
[GitHub] [airflow] kaxil commented on a change in pull request #18282: Refactor installation pages
kaxil commented on a change in pull request #18282:
URL: https://github.com/apache/airflow/pull/18282#discussion_r710520015
##########
File path: docs/apache-airflow/installation/index.rst
##########
@@ -0,0 +1,316 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+Installation
+------------
+
+.. contents:: :local:
+
+.. toctree::
+ :maxdepth: 1
+ :caption: Installation
+ :hidden:
+
+ Prerequisites <prerequisites>
+ Dependencies <dependencies>
+ Supported versions <supported-versions>
+ Installing from sources <installing-from-sources>
+ Installing from PyPI <installing-from-pypi>
+ Setting up the database <setting-up-the-database>
+ Upgrading to Airflow 2 <upgrading-to-2>
+ Upgrade Check <upgrade-check>
+
+This page describes installations options that you might use when considering how to install Airflow.
+Airflow consists of many components, often distributed among many physical or virtual machines, therefore
+installation of Airflow might be quite complex, depending on the options you choose.
+
+You should also check-out the :doc:`Prerequisites <prerequisites>` that must be fulfilled when installing Airflow
+as well as :doc:`Supported versions <supported-versions>` to know what are the policies for supporting
+Airflow, Python and Kubernetes.
+
+Airflow requires additional :doc:`Dependencies <dependencies>` to be installed - which can be done
+via extras and providers.
+
+When you install Airflow, you need to :doc:`setup the database <setting-up-the-database>` which must be
+kept updated when airflow is upgraded.
+
+If you came here to upgrade Airflow from 1.10 version (which is end-of-life already), follow the
+:doc:`Upgrading to Airflow 2 <upgrading-to-2>`.
+
+Using released sources
+''''''''''''''''''''''
+
+More details: :doc:`installing-from-sources`
+
+**When this option works best**
+
+* This option is best if you expect to build all your software from sources.
+* Apache Airflow is one of the projects that belong to the `Apache Software Foundation <https://www.apache.org/>`__ .
+ It is a requirement for all ASF projects that they can be installed using official sources released via `Official Apache Mirrors <http://ws.apache.org/mirrors.cgi/>`__ .
+* This is the best choice you have strong need to `Verify the integrity and provenance of the software <https://www.apache.org/dyn/closer.cgi#verify>`__
Review comment:
```suggestion
* This is the best choice if you have a strong need to `verify the integrity and provenance of the software <https://www.apache.org/dyn/closer.cgi#verify>`__
```
##########
File path: docs/apache-airflow/installation/index.rst
##########
@@ -0,0 +1,316 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+Installation
+------------
+
+.. contents:: :local:
+
+.. toctree::
+ :maxdepth: 1
+ :caption: Installation
+ :hidden:
+
+ Prerequisites <prerequisites>
+ Dependencies <dependencies>
+ Supported versions <supported-versions>
+ Installing from sources <installing-from-sources>
+ Installing from PyPI <installing-from-pypi>
+ Setting up the database <setting-up-the-database>
+ Upgrading to Airflow 2 <upgrading-to-2>
+ Upgrade Check <upgrade-check>
+
+This page describes installations options that you might use when considering how to install Airflow.
+Airflow consists of many components, often distributed among many physical or virtual machines, therefore
+installation of Airflow might be quite complex, depending on the options you choose.
+
+You should also check-out the :doc:`Prerequisites <prerequisites>` that must be fulfilled when installing Airflow
+as well as :doc:`Supported versions <supported-versions>` to know what are the policies for supporting
+Airflow, Python and Kubernetes.
+
+Airflow requires additional :doc:`Dependencies <dependencies>` to be installed - which can be done
+via extras and providers.
+
+When you install Airflow, you need to :doc:`setup the database <setting-up-the-database>` which must be
+kept updated when airflow is upgraded.
+
+If you came here to upgrade Airflow from 1.10 version (which is end-of-life already), follow the
+:doc:`Upgrading to Airflow 2 <upgrading-to-2>`.
+
+Using released sources
+''''''''''''''''''''''
+
+More details: :doc:`installing-from-sources`
+
+**When this option works best**
+
+* This option is best if you expect to build all your software from sources.
+* Apache Airflow is one of the projects that belong to the `Apache Software Foundation <https://www.apache.org/>`__ .
+ It is a requirement for all ASF projects that they can be installed using official sources released via `Official Apache Mirrors <http://ws.apache.org/mirrors.cgi/>`__ .
+* This is the best choice you have strong need to `Verify the integrity and provenance of the software <https://www.apache.org/dyn/closer.cgi#verify>`__
+
+**Intended users**
+
+* Users who are familiar with installing and building software from sources and are conscious about integrity and provenance
+ of the software they use down to the lowest level possible.
+
+**What are you expected to handle**
+
+* You are expected to build and install airflow and its components on your own.
+* You should develop and handle the deployment for all components of Airflow.
+* You are responsible for setting up database, creating and managing database schema with ``airflow db`` commands,
+ automated startup and recovery, maintenance, cleanup and upgrades of Airflow and the Airflow Providers.
+
+**What Apache Airflow Community provides for that method**
+
+* You have `instructions <https://github.com/apache/airflow/blob/main/INSTALL>`__ on how to build the software but due to various environments
+ and tools you might want to use, you might expect that there will be problems which are specific to your deployment and environment
+ you will have to diagnose and solve.
+
+**Where to ask for help**
+
+* The ``#development`` slack channel for building the software.
+
+* The ``#troubleshooting`` slack is a channel for quick general troubleshooting questions. The
+ `GitHub discussions <https://github.com/apache/airflow/discussions>`__ if you look for longer discussion and have more information to share.
+
+* If you can provide description of a reproducible problem with Airflow software, you can open issue at `GitHub issues <https://github.com/apache/airflow/issues>`_
+
+Using PyPI
+'''''''''''
+
+More details: :doc:`/installation/installing-from-pypi`
+
+**When this option works best**
+
+* This installation method is useful when you are not familiar with Containers and Docker and want to install
+ Apache Airflow on physical or virtual machines and you are used to installing and running software using custom
+ deployment mechanism.
+
+* The only officially supported mechanism of installation is via ``pip`` using constraint mechanisms. The constraint
+ files are managed by Apache Airflow release managers to make sure that you can repeatably install Airflow from PyPI with all Providers and
+ required dependencies.
+
+* In case of PyPI installation you could also verify integrity and provenance of the packages of the packages
+ downloaded from PyPI as described at the installation page, but software you download from PyPI is pre-built
+ for you so that you can install it without building, and you do not build the software from sources.
+
+**Intended users**
+
+* Users who are familiar with installing and configuring Python applications, managing Python environments,
+ dependencies and running software with their custom deployment mechanisms.
+
+**What are you expected to handle**
+
+* You are expected to install Airflow - all components of it - on your own.
+* You should develop and handle the deployment for all components of Airflow.
+* You are responsible for setting up database, creating and managing database schema with ``airflow db`` commands,
+ automated startup and recovery, maintenance, cleanup and upgrades of Airflow and Airflow Providers.
+
+**What Apache Airflow Community provides for that method**
+
+* You have :doc:`/installation/installing-from-pypi`
+ on how to install the software but due to various environments and tools you might want to use, you might
+ expect that there will be problems which are specific to your deployment and environment you will have to
+ diagnose and solve.
+* You have :doc:`/start/local` where you can see an example of Quick Start with running Airflow
+ locally which you can use to start Airflow quickly for local testing and development.
+ However this is just an inspiration. Do not expect this docker-compose is ready for production installation,
+ you need to build your own production-ready deployment in this approach.
+
+**Where to ask for help**
+
+* The ``#troubleshooting`` channel on Airflow Slack for quick general
+ troubleshooting questions. The `GitHub discussions <https://github.com/apache/airflow/discussions>`__
+ if you look for longer discussion and have more information to share.
+* If you can provide description of a reproducible problem with Airflow software, you can open
+ issue at `GitHub issues <https://github.com/apache/airflow/issues>`__
+
+
+Using Production Docker Images
+''''''''''''''''''''''''''''''
+
+More details: :doc:`docker-stack:index`
+
+**When this option works best**
+
+This installation method is useful when you are familiar with Container/Docker stack. It provides a capability of
+running Airflow components in isolation from other software running on the same physical or virtual machines with easy
+maintenance of dependencies.
+
+The images are build by Apache Airflow release managers and they use officially released packages from PyPI
+and official constraint files- same that are used for installing Airflow from PyPI.
Review comment:
I feel all the "When this options works best", "Intended Users", "Expected to handle" can be summarized in a table instead of separate paragraph so it is easy to compare, or a separate section on "Which method of Installation should be used?"
##########
File path: docs/apache-airflow/installation/index.rst
##########
@@ -0,0 +1,316 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+Installation
+------------
+
+.. contents:: :local:
+
+.. toctree::
+ :maxdepth: 1
+ :caption: Installation
+ :hidden:
+
+ Prerequisites <prerequisites>
+ Dependencies <dependencies>
+ Supported versions <supported-versions>
+ Installing from sources <installing-from-sources>
+ Installing from PyPI <installing-from-pypi>
+ Setting up the database <setting-up-the-database>
+ Upgrading to Airflow 2 <upgrading-to-2>
+ Upgrade Check <upgrade-check>
Review comment:
This feels like a separation Upgrading section instead of installation, wdyt?
##########
File path: docs/apache-airflow/installation/dependencies.rst
##########
@@ -0,0 +1,93 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Dependencies
+------------
+
+Airflow extra dependencies
+''''''''''''''''''''''''''
+
+The ``apache-airflow`` PyPI basic package only installs what's needed to get started.
+Additional packages can be installed depending on what will be useful in your
+environment. For instance, if you don't need connectivity with Postgres,
+you won't have to go through the trouble of installing the ``postgres-devel``
+yum package, or whatever equivalent applies on the distribution you are using.
+
+Most of the extra dependencies are linked to a corresponding provider package. For example "amazon" extra
+has a corresponding ``apache-airflow-providers-amazon`` provider package to be installed. When you install
+Airflow with such extras, the necessary provider packages are installed automatically (latest versions from
+PyPI for those packages). However you can freely upgrade and install provider packages independently from
+the main Airflow installation.
+
+For the list of the extras and what they enable, see: :doc:`/extra-packages-ref`.
+
+Provider packages
+'''''''''''''''''
+
+Unlike Apache Airflow 1.10, the Airflow 2.0 is delivered in multiple, separate, but connected packages.
+The core of Airflow scheduling system is delivered as ``apache-airflow`` package and there are around
+60 provider packages which can be installed separately as so called ``Airflow Provider packages``.
+The default Airflow installation doesn't have many integrations and you have to install them yourself.
+
+You can even develop and install your own providers for Airflow. For more information,
+see: :doc:`apache-airflow-providers:index`
+
+For the list of the provider packages and what they enable, see: :doc:`apache-airflow-providers:packages-ref`.
+
+Differences between extras and providers
+''''''''''''''''''''''''''''''''''''''''
+
+Just to prevent confusion of extras versus provider packages: Extras and providers are different things,
+though many extras are leading to installing providers.
+
+Extras are standard Python setuptools feature that allows to add additional set of dependencies as
+optional features to "core" Apache Airflow. One of the type of such optional features are providers
+packages, but not all optional features of Apache Airflow have corresponding providers.
+
+We are using the ``extras`` setuptools features to also install provider packages.
+Most of the extras are also linked (same name) with provider packages - for example adding ``[google]``
+extra also adds ``apache-airflow-providers-google`` as dependency. However there are some extras that do
+not install providers (examples ``github_enterprise``, ``kerberos``, ``async`` - they add some extra
+dependencies which are needed for those ``extra`` features of Airflow mentioned. The three examples
+above add respectively github enterprise oauth authentication, kerberos integration or
+asynchronous workers for gunicorn. None of those have providers, they are just extending Apache Airflow
+"core" package with new functionalities.
+
+System dependencies
+'''''''''''''''''''
+
+You need certain system level requirements in order to install Airflow. Those are requirements that are known
+to be needed for Linux system (Tested on Ubuntu Buster LTS) :
+
+.. code-block:: bash
+ :substitutions:
Review comment:
```suggestion
```
Since we don't substitute anything
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org