You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2018/05/23 09:08:54 UTC
incubator-airflow git commit: [AIRFLOW-2509] Separate config docs
into how-to guides
Repository: incubator-airflow
Updated Branches:
refs/heads/master d52e9e664 -> 084bc9136
[AIRFLOW-2509] Separate config docs into how-to guides
Also moves how-to style instructions for logging
from "integration" page
to a "Writing Logs" how-to.
Closes #3400 from tswast/howto
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/084bc913
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/084bc913
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/084bc913
Branch: refs/heads/master
Commit: 084bc913674218efbac6ee354c241b01726f1e05
Parents: d52e9e6
Author: Tim Swast <sw...@google.com>
Authored: Wed May 23 10:08:53 2018 +0100
Committer: Kaxil Naik <ka...@apache.org>
Committed: Wed May 23 10:08:53 2018 +0100
----------------------------------------------------------------------
docs/configuration.rst | 348 --------------------------------
docs/howto/create-connection.rst | 8 +
docs/howto/executor/use-celery.rst | 47 +++++
docs/howto/executor/use-dask.rst | 33 +++
docs/howto/executor/use-mesos.rst | 65 ++++++
docs/howto/index.rst | 23 +++
docs/howto/initialize-database.rst | 31 +++
docs/howto/run-with-systemd.rst | 15 ++
docs/howto/run-with-upstart.rst | 20 ++
docs/howto/secure-connections.rst | 39 ++++
docs/howto/set-config.rst | 35 ++++
docs/howto/use-test-config.rst | 17 ++
docs/howto/write-logs.rst | 159 +++++++++++++++
docs/index.rst | 2 +-
docs/installation.rst | 17 +-
docs/integration.rst | 99 +--------
docs/start.rst | 2 +-
17 files changed, 507 insertions(+), 453 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/configuration.rst
----------------------------------------------------------------------
diff --git a/docs/configuration.rst b/docs/configuration.rst
deleted file mode 100644
index ea561b9..0000000
--- a/docs/configuration.rst
+++ /dev/null
@@ -1,348 +0,0 @@
-Configuration
--------------
-
-Setting up the sandbox in the :doc:`start` section was easy;
-building a production-grade environment requires a bit more work!
-
-.. _setting-options:
-
-Setting Configuration Options
-'''''''''''''''''''''''''''''
-
-The first time you run Airflow, it will create a file called ``airflow.cfg`` in
-your ``$AIRFLOW_HOME`` directory (``~/airflow`` by default). This file contains Airflow's configuration and you
-can edit it to change any of the settings. You can also set options with environment variables by using this format:
-``$AIRFLOW__{SECTION}__{KEY}`` (note the double underscores).
-
-For example, the
-metadata database connection string can either be set in ``airflow.cfg`` like this:
-
-.. code-block:: bash
-
- [core]
- sql_alchemy_conn = my_conn_string
-
-or by creating a corresponding environment variable:
-
-.. code-block:: bash
-
- AIRFLOW__CORE__SQL_ALCHEMY_CONN=my_conn_string
-
-You can also derive the connection string at run time by appending ``_cmd`` to the key like this:
-
-.. code-block:: bash
-
- [core]
- sql_alchemy_conn_cmd = bash_command_to_run
-
--But only three such configuration elements namely sql_alchemy_conn, broker_url and result_backend can be fetched as a command. The idea behind this is to not store passwords on boxes in plain text files. The order of precedence is as follows -
-
-1. environment variable
-2. configuration in airflow.cfg
-3. command in airflow.cfg
-4. default
-
-Setting up a Backend
-''''''''''''''''''''
-If you want to take a real test drive of Airflow, you should consider
-setting up a real database backend and switching to the LocalExecutor.
-
-As Airflow was built to interact with its metadata using the great SqlAlchemy
-library, you should be able to use any database backend supported as a
-SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
-
-.. note:: We rely on more strict ANSI SQL settings for MySQL in order to have
- sane defaults. Make sure to have specified `explicit_defaults_for_timestamp=1`
- in your my.cnf under `[mysqld]`
-
-.. note:: If you decide to use **Postgres**, we recommend using the ``psycopg2``
- driver and specifying it in your SqlAlchemy connection string.
- Also note that since SqlAlchemy does not expose a way to target a
- specific schema in the Postgres connection URI, you may
- want to set a default schema for your role with a
- command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
-
-Once you've setup your database to host Airflow, you'll need to alter the
-SqlAlchemy connection string located in your configuration file
-``$AIRFLOW_HOME/airflow.cfg``. You should then also change the "executor"
-setting to use "LocalExecutor", an executor that can parallelize task
-instances locally.
-
-.. code-block:: bash
-
- # initialize the database
- airflow initdb
-
-Connections
-'''''''''''
-Airflow needs to know how to connect to your environment. Information
-such as hostname, port, login and passwords to other systems and services is
-handled in the ``Admin->Connection`` section of the UI. The pipeline code you
-will author will reference the 'conn_id' of the Connection objects.
-
-.. image:: img/connections.png
-
-By default, Airflow will save the passwords for the connection in plain text
-within the metadata database. The ``crypto`` package is highly recommended
-during installation. The ``crypto`` package does require that your operating
-system have libffi-dev installed.
-
-If ``crypto`` package was not installed initially, you can still enable encryption for
-connections by following steps below:
-
-1. Install crypto package ``pip install apache-airflow[crypto]``
-2. Generate fernet_key, using this code snippet below. fernet_key must be a base64-encoded 32-byte key.
-
-.. code:: python
-
- from cryptography.fernet import Fernet
- fernet_key= Fernet.generate_key()
- print(fernet_key) # your fernet_key, keep it in secured place!
-
-3. Replace ``airflow.cfg`` fernet_key value with the one from step 2.
-Alternatively, you can store your fernet_key in OS environment variable. You
-do not need to change ``airflow.cfg`` in this case as Airflow will use environment
-variable over the value in ``airflow.cfg``:
-
-.. code-block:: bash
-
- # Note the double underscores
- EXPORT AIRFLOW__CORE__FERNET_KEY = your_fernet_key
-
-4. Restart Airflow webserver.
-5. For existing connections (the ones that you had defined before installing ``airflow[crypto]`` and creating a Fernet key), you need to open each connection in the connection admin UI, re-type the password, and save it.
-
-Connections in Airflow pipelines can be created using environment variables.
-The environment variable needs to have a prefix of ``AIRFLOW_CONN_`` for
-Airflow with the value in a URI format to use the connection properly. Please
-see the :doc:`concepts` documentation for more information on environment
-variables and connections.
-
-Scaling Out with Celery
-'''''''''''''''''''''''
-``CeleryExecutor`` is one of the ways you can scale out the number of workers. For this
-to work, you need to setup a Celery backend (**RabbitMQ**, **Redis**, ...) and
-change your ``airflow.cfg`` to point the executor parameter to
-``CeleryExecutor`` and provide the related Celery settings.
-
-For more information about setting up a Celery broker, refer to the
-exhaustive `Celery documentation on the topic <http://docs.celeryproject.org/en/latest/getting-started/brokers/index.html>`_.
-
-Here are a few imperative requirements for your workers:
-
-- ``airflow`` needs to be installed, and the CLI needs to be in the path
-- Airflow configuration settings should be homogeneous across the cluster
-- Operators that are executed on the worker need to have their dependencies
- met in that context. For example, if you use the ``HiveOperator``,
- the hive CLI needs to be installed on that box, or if you use the
- ``MySqlOperator``, the required Python library needs to be available in
- the ``PYTHONPATH`` somehow
-- The worker needs to have access to its ``DAGS_FOLDER``, and you need to
- synchronize the filesystems by your own means. A common setup would be to
- store your DAGS_FOLDER in a Git repository and sync it across machines using
- Chef, Puppet, Ansible, or whatever you use to configure machines in your
- environment. If all your boxes have a common mount point, having your
- pipelines files shared there should work as well
-
-
-To kick off a worker, you need to setup Airflow and kick off the worker
-subcommand
-
-.. code-block:: bash
-
- airflow worker
-
-Your worker should start picking up tasks as soon as they get fired in
-its direction.
-
-Note that you can also run "Celery Flower", a web UI built on top of Celery,
-to monitor your workers. You can use the shortcut command ``airflow flower``
-to start a Flower web server.
-
-Some caveats:
-
-- Make sure to use a database backed result backend
-- Make sure to set a visibility timeout in [celery_broker_transport_options] that exceeds the ETA of your longest running task
-- Tasks can and consume resources, make sure your worker as enough resources to run `worker_concurrency` tasks
-
-Scaling Out with Dask
-'''''''''''''''''''''
-
-``DaskExecutor`` allows you to run Airflow tasks in a Dask Distributed cluster.
-
-Dask clusters can be run on a single machine or on remote networks. For complete
-details, consult the `Distributed documentation <https://distributed.readthedocs.io/>`_.
-
-To create a cluster, first start a Scheduler:
-
-.. code-block:: bash
-
- # default settings for a local cluster
- DASK_HOST=127.0.0.1
- DASK_PORT=8786
-
- dask-scheduler --host $DASK_HOST --port $DASK_PORT
-
-Next start at least one Worker on any machine that can connect to the host:
-
-.. code-block:: bash
-
- dask-worker $DASK_HOST:$DASK_PORT
-
-Edit your ``airflow.cfg`` to set your executor to ``DaskExecutor`` and provide
-the Dask Scheduler address in the ``[dask]`` section.
-
-Please note:
-
-- Each Dask worker must be able to import Airflow and any dependencies you
- require.
-- Dask does not support queues. If an Airflow task was created with a queue, a
- warning will be raised but the task will be submitted to the cluster.
-
-
-Logs
-''''
-Users can specify a logs folder in ``airflow.cfg``. By default, it is in
-the ``AIRFLOW_HOME`` directory.
-
-In addition, users can supply a remote location for storing logs and log backups
-in cloud storage. At this time, Amazon S3 and Google Cloud Storage are supported.
-To enable this feature, ``airflow.cfg`` must be configured as in this example:
-
-.. code-block:: bash
-
- [core]
- # Airflow can store logs remotely in AWS S3 or Google Cloud Storage. Users
- # must supply a remote location URL (starting with either 's3://...' or
- # 'gs://...') and an Airflow connection id that provides access to the storage
- # location.
- remote_base_log_folder = s3://my-bucket/path/to/logs
- remote_log_conn_id = MyS3Conn
- # Use server-side encryption for logs stored in S3
- encrypt_s3_logs = False
-
-Remote logging uses an existing Airflow connection to read/write logs. If you don't
-have a connection properly setup, this will fail. In the above example, Airflow will
-try to use ``S3Hook('MyS3Conn')``.
-
-In the Airflow Web UI, local logs take precedance over remote logs. If local logs
-can not be found or accessed, the remote logs will be displayed. Note that logs
-are only sent to remote storage once a task completes (including failure). In other
-words, remote logs for running tasks are unavailable. Logs are stored in the log
-folder as ``{dag_id}/{task_id}/{execution_date}/{try_number}.log``.
-
-Scaling Out on Mesos (community contributed)
-''''''''''''''''''''''''''''''''''''''''''''
-There are two ways you can run airflow as a mesos framework:
-
-1. Running airflow tasks directly on mesos slaves, requiring each mesos slave to have airflow installed and configured.
-2. Running airflow tasks inside a docker container that has airflow installed, which is run on a mesos slave.
-
-Tasks executed directly on mesos slaves
-=======================================
-``MesosExecutor`` allows you to schedule airflow tasks on a Mesos cluster.
-For this to work, you need a running mesos cluster and you must perform the following
-steps -
-
-1. Install airflow on a mesos slave where web server and scheduler will run,
- let's refer to this as the "Airflow server".
-2. On the Airflow server, install mesos python eggs from `mesos downloads <http://open.mesosphere.com/downloads/mesos/>`_.
-3. On the Airflow server, use a database (such as mysql) which can be accessed from all mesos
- slaves and add configuration in ``airflow.cfg``.
-4. Change your ``airflow.cfg`` to point executor parameter to
- `MesosExecutor` and provide related Mesos settings.
-5. On all mesos slaves, install airflow. Copy the ``airflow.cfg`` from
- Airflow server (so that it uses same sql alchemy connection).
-6. On all mesos slaves, run the following for serving logs:
-
-.. code-block:: bash
-
- airflow serve_logs
-
-7. On Airflow server, to start processing/scheduling DAGs on mesos, run:
-
-.. code-block:: bash
-
- airflow scheduler -p
-
-Note: We need -p parameter to pickle the DAGs.
-
-You can now see the airflow framework and corresponding tasks in mesos UI.
-The logs for airflow tasks can be seen in airflow UI as usual.
-
-For more information about mesos, refer to `mesos documentation <http://mesos.apache.org/documentation/latest/>`_.
-For any queries/bugs on `MesosExecutor`, please contact `@kapil-malik <https://github.com/kapil-malik>`_.
-
-Tasks executed in containers on mesos slaves
-============================================
-
-`This gist <https://gist.github.com/sebradloff/f158874e615bda0005c6f4577b20036e>`_ contains all files and configuration changes necessary to achieve the following:
-
-1. Create a dockerized version of airflow with mesos python eggs installed.
-
- We recommend taking advantage of docker's multi stage builds in order to achieve this. We have one Dockerfile that defines building a specific version of mesos from source (Dockerfile-mesos), in order to create the python eggs. In the airflow Dockerfile (Dockerfile-airflow) we copy the python eggs from the mesos image.
-
-2. Create a mesos configuration block within the ``airflow.cfg``.
-
- The configuration block remains the same as the default airflow configuration (default_airflow.cfg), but has the addition of an option ``docker_image_slave``. This should be set to the name of the image you would like mesos to use when running airflow tasks. Make sure you have the proper configuration of the DNS record for your mesos master and any sort of authorization if any exists.
-
-3. Change your ``airflow.cfg`` to point the executor parameter to
- `MesosExecutor` (`executor = SequentialExecutor`).
-
-4. Make sure your mesos slave has access to the docker repository you are using for your ``docker_image_slave``.
-
- `Instructions are available in the mesos docs. <https://mesos.readthedocs.io/en/latest/docker-containerizer/#private-docker-repository>`_
-
-The rest is up to you and how you want to work with a dockerized airflow configuration.
-
-Integration with systemd
-''''''''''''''''''''''''
-Airflow can integrate with systemd based systems. This makes watching your
-daemons easy as systemd can take care of restarting a daemon on failure.
-In the ``scripts/systemd`` directory you can find unit files that
-have been tested on Redhat based systems. You can copy those to
-``/usr/lib/systemd/system``. It is assumed that Airflow will run under
-``airflow:airflow``. If not (or if you are running on a non Redhat
-based system) you probably need to adjust the unit files.
-
-Environment configuration is picked up from ``/etc/sysconfig/airflow``.
-An example file is supplied. Make sure to specify the ``SCHEDULER_RUNS``
-variable in this file when you run the scheduler. You
-can also define here, for example, ``AIRFLOW_HOME`` or ``AIRFLOW_CONFIG``.
-
-Integration with upstart
-''''''''''''''''''''''''
-Airflow can integrate with upstart based systems. Upstart automatically starts all airflow services for which you
-have a corresponding ``*.conf`` file in ``/etc/init`` upon system boot. On failure, upstart automatically restarts
-the process (until it reaches re-spawn limit set in a ``*.conf`` file).
-
-You can find sample upstart job files in the ``scripts/upstart`` directory. These files have been tested on
-Ubuntu 14.04 LTS. You may have to adjust ``start on`` and ``stop on`` stanzas to make it work on other upstart
-systems. Some of the possible options are listed in ``scripts/upstart/README``.
-
-Modify ``*.conf`` files as needed and copy to ``/etc/init`` directory. It is assumed that airflow will run
-under ``airflow:airflow``. Change ``setuid`` and ``setgid`` in ``*.conf`` files if you use other user/group
-
-You can use ``initctl`` to manually start, stop, view status of the airflow process that has been
-integrated with upstart
-
-.. code-block:: bash
-
- initctl airflow-webserver status
-
-Test Mode
-'''''''''
-Airflow has a fixed set of "test mode" configuration options. You can load these
-at any time by calling ``airflow.configuration.load_test_config()`` (note this
-operation is not reversible!). However, some options (like the DAG_FOLDER) are
-loaded before you have a chance to call load_test_config(). In order to eagerly load
-the test configuration, set test_mode in airflow.cfg:
-
-.. code-block:: bash
-
- [tests]
- unit_test_mode = True
-
-Due to Airflow's automatic environment variable expansion (see :ref:`setting-options`),
-you can also set the env var ``AIRFLOW__CORE__UNIT_TEST_MODE`` to temporarily overwrite
-airflow.cfg.
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/howto/create-connection.rst
----------------------------------------------------------------------
diff --git a/docs/howto/create-connection.rst b/docs/howto/create-connection.rst
new file mode 100644
index 0000000..ba9f444
--- /dev/null
+++ b/docs/howto/create-connection.rst
@@ -0,0 +1,8 @@
+Creating a Connection
+=====================
+
+Connections in Airflow pipelines can be created using environment variables.
+The environment variable needs to have a prefix of ``AIRFLOW_CONN_`` for
+Airflow with the value in a URI format to use the connection properly. Please
+see the :doc:`../../concepts` documentation for more information on environment
+variables and connections.
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/howto/executor/use-celery.rst
----------------------------------------------------------------------
diff --git a/docs/howto/executor/use-celery.rst b/docs/howto/executor/use-celery.rst
new file mode 100644
index 0000000..fd6db96
--- /dev/null
+++ b/docs/howto/executor/use-celery.rst
@@ -0,0 +1,47 @@
+Scaling Out with Celery
+=======================
+
+``CeleryExecutor`` is one of the ways you can scale out the number of workers. For this
+to work, you need to setup a Celery backend (**RabbitMQ**, **Redis**, ...) and
+change your ``airflow.cfg`` to point the executor parameter to
+``CeleryExecutor`` and provide the related Celery settings.
+
+For more information about setting up a Celery broker, refer to the
+exhaustive `Celery documentation on the topic <http://docs.celeryproject.org/en/latest/getting-started/brokers/index.html>`_.
+
+Here are a few imperative requirements for your workers:
+
+- ``airflow`` needs to be installed, and the CLI needs to be in the path
+- Airflow configuration settings should be homogeneous across the cluster
+- Operators that are executed on the worker need to have their dependencies
+ met in that context. For example, if you use the ``HiveOperator``,
+ the hive CLI needs to be installed on that box, or if you use the
+ ``MySqlOperator``, the required Python library needs to be available in
+ the ``PYTHONPATH`` somehow
+- The worker needs to have access to its ``DAGS_FOLDER``, and you need to
+ synchronize the filesystems by your own means. A common setup would be to
+ store your DAGS_FOLDER in a Git repository and sync it across machines using
+ Chef, Puppet, Ansible, or whatever you use to configure machines in your
+ environment. If all your boxes have a common mount point, having your
+ pipelines files shared there should work as well
+
+
+To kick off a worker, you need to setup Airflow and kick off the worker
+subcommand
+
+.. code-block:: bash
+
+ airflow worker
+
+Your worker should start picking up tasks as soon as they get fired in
+its direction.
+
+Note that you can also run "Celery Flower", a web UI built on top of Celery,
+to monitor your workers. You can use the shortcut command ``airflow flower``
+to start a Flower web server.
+
+Some caveats:
+
+- Make sure to use a database backed result backend
+- Make sure to set a visibility timeout in [celery_broker_transport_options] that exceeds the ETA of your longest running task
+- Tasks can and consume resources, make sure your worker as enough resources to run `worker_concurrency` tasks
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/howto/executor/use-dask.rst
----------------------------------------------------------------------
diff --git a/docs/howto/executor/use-dask.rst b/docs/howto/executor/use-dask.rst
new file mode 100644
index 0000000..769ce17
--- /dev/null
+++ b/docs/howto/executor/use-dask.rst
@@ -0,0 +1,33 @@
+Scaling Out with Dask
+=====================
+
+``DaskExecutor`` allows you to run Airflow tasks in a Dask Distributed cluster.
+
+Dask clusters can be run on a single machine or on remote networks. For complete
+details, consult the `Distributed documentation <https://distributed.readthedocs.io/>`_.
+
+To create a cluster, first start a Scheduler:
+
+.. code-block:: bash
+
+ # default settings for a local cluster
+ DASK_HOST=127.0.0.1
+ DASK_PORT=8786
+
+ dask-scheduler --host $DASK_HOST --port $DASK_PORT
+
+Next start at least one Worker on any machine that can connect to the host:
+
+.. code-block:: bash
+
+ dask-worker $DASK_HOST:$DASK_PORT
+
+Edit your ``airflow.cfg`` to set your executor to ``DaskExecutor`` and provide
+the Dask Scheduler address in the ``[dask]`` section.
+
+Please note:
+
+- Each Dask worker must be able to import Airflow and any dependencies you
+ require.
+- Dask does not support queues. If an Airflow task was created with a queue, a
+ warning will be raised but the task will be submitted to the cluster.
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/howto/executor/use-mesos.rst
----------------------------------------------------------------------
diff --git a/docs/howto/executor/use-mesos.rst b/docs/howto/executor/use-mesos.rst
new file mode 100644
index 0000000..c3bf95a
--- /dev/null
+++ b/docs/howto/executor/use-mesos.rst
@@ -0,0 +1,65 @@
+Scaling Out with Mesos (community contributed)
+==============================================
+
+There are two ways you can run airflow as a mesos framework:
+
+1. Running airflow tasks directly on mesos slaves, requiring each mesos slave to have airflow installed and configured.
+2. Running airflow tasks inside a docker container that has airflow installed, which is run on a mesos slave.
+
+Tasks executed directly on mesos slaves
+---------------------------------------
+
+``MesosExecutor`` allows you to schedule airflow tasks on a Mesos cluster.
+For this to work, you need a running mesos cluster and you must perform the following
+steps -
+
+1. Install airflow on a mesos slave where web server and scheduler will run,
+ let's refer to this as the "Airflow server".
+2. On the Airflow server, install mesos python eggs from `mesos downloads <http://open.mesosphere.com/downloads/mesos/>`_.
+3. On the Airflow server, use a database (such as mysql) which can be accessed from all mesos
+ slaves and add configuration in ``airflow.cfg``.
+4. Change your ``airflow.cfg`` to point executor parameter to
+ `MesosExecutor` and provide related Mesos settings.
+5. On all mesos slaves, install airflow. Copy the ``airflow.cfg`` from
+ Airflow server (so that it uses same sql alchemy connection).
+6. On all mesos slaves, run the following for serving logs:
+
+.. code-block:: bash
+
+ airflow serve_logs
+
+7. On Airflow server, to start processing/scheduling DAGs on mesos, run:
+
+.. code-block:: bash
+
+ airflow scheduler -p
+
+Note: We need -p parameter to pickle the DAGs.
+
+You can now see the airflow framework and corresponding tasks in mesos UI.
+The logs for airflow tasks can be seen in airflow UI as usual.
+
+For more information about mesos, refer to `mesos documentation <http://mesos.apache.org/documentation/latest/>`_.
+For any queries/bugs on `MesosExecutor`, please contact `@kapil-malik <https://github.com/kapil-malik>`_.
+
+Tasks executed in containers on mesos slaves
+--------------------------------------------
+
+`This gist <https://gist.github.com/sebradloff/f158874e615bda0005c6f4577b20036e>`_ contains all files and configuration changes necessary to achieve the following:
+
+1. Create a dockerized version of airflow with mesos python eggs installed.
+
+ We recommend taking advantage of docker's multi stage builds in order to achieve this. We have one Dockerfile that defines building a specific version of mesos from source (Dockerfile-mesos), in order to create the python eggs. In the airflow Dockerfile (Dockerfile-airflow) we copy the python eggs from the mesos image.
+
+2. Create a mesos configuration block within the ``airflow.cfg``.
+
+ The configuration block remains the same as the default airflow configuration (default_airflow.cfg), but has the addition of an option ``docker_image_slave``. This should be set to the name of the image you would like mesos to use when running airflow tasks. Make sure you have the proper configuration of the DNS record for your mesos master and any sort of authorization if any exists.
+
+3. Change your ``airflow.cfg`` to point the executor parameter to
+ `MesosExecutor` (`executor = SequentialExecutor`).
+
+4. Make sure your mesos slave has access to the docker repository you are using for your ``docker_image_slave``.
+
+ `Instructions are available in the mesos docs. <https://mesos.readthedocs.io/en/latest/docker-containerizer/#private-docker-repository>`_
+
+The rest is up to you and how you want to work with a dockerized airflow configuration.
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/howto/index.rst
----------------------------------------------------------------------
diff --git a/docs/howto/index.rst b/docs/howto/index.rst
new file mode 100644
index 0000000..5c22a5d
--- /dev/null
+++ b/docs/howto/index.rst
@@ -0,0 +1,23 @@
+How-to Guides
+=============
+
+Setting up the sandbox in the :doc:`../start` section was easy;
+building a production-grade environment requires a bit more work!
+
+These how-to guides will step you through common tasks in using and
+configuring an Airflow environment.
+
+.. toctree::
+ :maxdepth: 2
+
+ set-config
+ initialize-database
+ secure-connections
+ create-connection
+ write-logs
+ executor/use-celery
+ executor/use-dask
+ executor/use-mesos
+ run-with-systemd
+ run-with-upstart
+ use-test-config
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/howto/initialize-database.rst
----------------------------------------------------------------------
diff --git a/docs/howto/initialize-database.rst b/docs/howto/initialize-database.rst
new file mode 100644
index 0000000..c85142f
--- /dev/null
+++ b/docs/howto/initialize-database.rst
@@ -0,0 +1,31 @@
+Initializing a Database Backend
+===============================
+
+If you want to take a real test drive of Airflow, you should consider
+setting up a real database backend and switching to the LocalExecutor.
+
+As Airflow was built to interact with its metadata using the great SqlAlchemy
+library, you should be able to use any database backend supported as a
+SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
+
+.. note:: We rely on more strict ANSI SQL settings for MySQL in order to have
+ sane defaults. Make sure to have specified `explicit_defaults_for_timestamp=1`
+ in your my.cnf under `[mysqld]`
+
+.. note:: If you decide to use **Postgres**, we recommend using the ``psycopg2``
+ driver and specifying it in your SqlAlchemy connection string.
+ Also note that since SqlAlchemy does not expose a way to target a
+ specific schema in the Postgres connection URI, you may
+ want to set a default schema for your role with a
+ command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+
+Once you've setup your database to host Airflow, you'll need to alter the
+SqlAlchemy connection string located in your configuration file
+``$AIRFLOW_HOME/airflow.cfg``. You should then also change the "executor"
+setting to use "LocalExecutor", an executor that can parallelize task
+instances locally.
+
+.. code-block:: bash
+
+ # initialize the database
+ airflow initdb
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/howto/run-with-systemd.rst
----------------------------------------------------------------------
diff --git a/docs/howto/run-with-systemd.rst b/docs/howto/run-with-systemd.rst
new file mode 100644
index 0000000..1dc1383
--- /dev/null
+++ b/docs/howto/run-with-systemd.rst
@@ -0,0 +1,15 @@
+Running Airflow with systemd
+============================
+
+Airflow can integrate with systemd based systems. This makes watching your
+daemons easy as systemd can take care of restarting a daemon on failure.
+In the ``scripts/systemd`` directory you can find unit files that
+have been tested on Redhat based systems. You can copy those to
+``/usr/lib/systemd/system``. It is assumed that Airflow will run under
+``airflow:airflow``. If not (or if you are running on a non Redhat
+based system) you probably need to adjust the unit files.
+
+Environment configuration is picked up from ``/etc/sysconfig/airflow``.
+An example file is supplied. Make sure to specify the ``SCHEDULER_RUNS``
+variable in this file when you run the scheduler. You
+can also define here, for example, ``AIRFLOW_HOME`` or ``AIRFLOW_CONFIG``.
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/howto/run-with-upstart.rst
----------------------------------------------------------------------
diff --git a/docs/howto/run-with-upstart.rst b/docs/howto/run-with-upstart.rst
new file mode 100644
index 0000000..c18a203
--- /dev/null
+++ b/docs/howto/run-with-upstart.rst
@@ -0,0 +1,20 @@
+Running Airflow with upstart
+============================
+
+Airflow can integrate with upstart based systems. Upstart automatically starts all airflow services for which you
+have a corresponding ``*.conf`` file in ``/etc/init`` upon system boot. On failure, upstart automatically restarts
+the process (until it reaches re-spawn limit set in a ``*.conf`` file).
+
+You can find sample upstart job files in the ``scripts/upstart`` directory. These files have been tested on
+Ubuntu 14.04 LTS. You may have to adjust ``start on`` and ``stop on`` stanzas to make it work on other upstart
+systems. Some of the possible options are listed in ``scripts/upstart/README``.
+
+Modify ``*.conf`` files as needed and copy to ``/etc/init`` directory. It is assumed that airflow will run
+under ``airflow:airflow``. Change ``setuid`` and ``setgid`` in ``*.conf`` files if you use other user/group
+
+You can use ``initctl`` to manually start, stop, view status of the airflow process that has been
+integrated with upstart
+
+.. code-block:: bash
+
+ initctl airflow-webserver status
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/howto/secure-connections.rst
----------------------------------------------------------------------
diff --git a/docs/howto/secure-connections.rst b/docs/howto/secure-connections.rst
new file mode 100644
index 0000000..5d468b3
--- /dev/null
+++ b/docs/howto/secure-connections.rst
@@ -0,0 +1,39 @@
+Securing Connections
+====================
+
+Airflow needs to know how to connect to your environment. Information
+such as hostname, port, login and passwords to other systems and services is
+handled in the ``Admin->Connection`` section of the UI. The pipeline code you
+will author will reference the 'conn_id' of the Connection objects.
+
+.. image:: ../img/connections.png
+
+By default, Airflow will save the passwords for the connection in plain text
+within the metadata database. The ``crypto`` package is highly recommended
+during installation. The ``crypto`` package does require that your operating
+system have libffi-dev installed.
+
+If ``crypto`` package was not installed initially, you can still enable encryption for
+connections by following steps below:
+
+1. Install crypto package ``pip install apache-airflow[crypto]``
+2. Generate fernet_key, using this code snippet below. fernet_key must be a base64-encoded 32-byte key.
+
+.. code:: python
+
+ from cryptography.fernet import Fernet
+ fernet_key= Fernet.generate_key()
+ print(fernet_key) # your fernet_key, keep it in secured place!
+
+3. Replace ``airflow.cfg`` fernet_key value with the one from step 2.
+Alternatively, you can store your fernet_key in OS environment variable. You
+do not need to change ``airflow.cfg`` in this case as Airflow will use environment
+variable over the value in ``airflow.cfg``:
+
+.. code-block:: bash
+
+ # Note the double underscores
+ EXPORT AIRFLOW__CORE__FERNET_KEY = your_fernet_key
+
+4. Restart Airflow webserver.
+5. For existing connections (the ones that you had defined before installing ``airflow[crypto]`` and creating a Fernet key), you need to open each connection in the connection admin UI, re-type the password, and save it.
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/howto/set-config.rst
----------------------------------------------------------------------
diff --git a/docs/howto/set-config.rst b/docs/howto/set-config.rst
new file mode 100644
index 0000000..2caf5d5
--- /dev/null
+++ b/docs/howto/set-config.rst
@@ -0,0 +1,35 @@
+Setting Configuration Options
+=============================
+
+The first time you run Airflow, it will create a file called ``airflow.cfg`` in
+your ``$AIRFLOW_HOME`` directory (``~/airflow`` by default). This file contains Airflow's configuration and you
+can edit it to change any of the settings. You can also set options with environment variables by using this format:
+``$AIRFLOW__{SECTION}__{KEY}`` (note the double underscores).
+
+For example, the
+metadata database connection string can either be set in ``airflow.cfg`` like this:
+
+.. code-block:: bash
+
+ [core]
+ sql_alchemy_conn = my_conn_string
+
+or by creating a corresponding environment variable:
+
+.. code-block:: bash
+
+ AIRFLOW__CORE__SQL_ALCHEMY_CONN=my_conn_string
+
+You can also derive the connection string at run time by appending ``_cmd`` to the key like this:
+
+.. code-block:: bash
+
+ [core]
+ sql_alchemy_conn_cmd = bash_command_to_run
+
+-But only three such configuration elements namely sql_alchemy_conn, broker_url and result_backend can be fetched as a command. The idea behind this is to not store passwords on boxes in plain text files. The order of precedence is as follows -
+
+1. environment variable
+2. configuration in airflow.cfg
+3. command in airflow.cfg
+4. default
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/howto/use-test-config.rst
----------------------------------------------------------------------
diff --git a/docs/howto/use-test-config.rst b/docs/howto/use-test-config.rst
new file mode 100644
index 0000000..5cb4790
--- /dev/null
+++ b/docs/howto/use-test-config.rst
@@ -0,0 +1,17 @@
+Using the Test Mode Configuration
+=================================
+
+Airflow has a fixed set of "test mode" configuration options. You can load these
+at any time by calling ``airflow.configuration.load_test_config()`` (note this
+operation is not reversible!). However, some options (like the DAG_FOLDER) are
+loaded before you have a chance to call load_test_config(). In order to eagerly load
+the test configuration, set test_mode in airflow.cfg:
+
+.. code-block:: bash
+
+ [tests]
+ unit_test_mode = True
+
+Due to Airflow's automatic environment variable expansion (see :doc:`set-config`),
+you can also set the env var ``AIRFLOW__CORE__UNIT_TEST_MODE`` to temporarily overwrite
+airflow.cfg.
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/howto/write-logs.rst
----------------------------------------------------------------------
diff --git a/docs/howto/write-logs.rst b/docs/howto/write-logs.rst
new file mode 100644
index 0000000..a29f606
--- /dev/null
+++ b/docs/howto/write-logs.rst
@@ -0,0 +1,159 @@
+Writing Logs
+============
+
+Writing Logs Locally
+--------------------
+
+Users can specify a logs folder in ``airflow.cfg`` using the
+``base_log_folder`` setting. By default, it is in the ``AIRFLOW_HOME``
+directory.
+
+In addition, users can supply a remote location for storing logs and log
+backups in cloud storage.
+
+In the Airflow Web UI, local logs take precedance over remote logs. If local logs
+can not be found or accessed, the remote logs will be displayed. Note that logs
+are only sent to remote storage once a task completes (including failure). In other
+words, remote logs for running tasks are unavailable. Logs are stored in the log
+folder as ``{dag_id}/{task_id}/{execution_date}/{try_number}.log``.
+
+.. _write-logs-amazon:
+
+Writing Logs to Amazon S3
+-------------------------
+
+Before you begin
+''''''''''''''''
+
+Remote logging uses an existing Airflow connection to read/write logs. If you
+don't have a connection properly setup, this will fail.
+
+Enabling remote logging
+'''''''''''''''''''''''
+
+To enable this feature, ``airflow.cfg`` must be configured as in this
+example:
+
+.. code-block:: bash
+
+ [core]
+ # Airflow can store logs remotely in AWS S3. Users must supply a remote
+ # location URL (starting with either 's3://...') and an Airflow connection
+ # id that provides access to the storage location.
+ remote_base_log_folder = s3://my-bucket/path/to/logs
+ remote_log_conn_id = MyS3Conn
+ # Use server-side encryption for logs stored in S3
+ encrypt_s3_logs = False
+
+In the above example, Airflow will try to use ``S3Hook('MyS3Conn')``.
+
+.. _write-logs-azure:
+
+Writing Logs to Azure Blob Storage
+----------------------------------
+
+Airflow can be configured to read and write task logs in Azure Blob Storage.
+Follow the steps below to enable Azure Blob Storage logging.
+
+#. Airflow's logging system requires a custom .py file to be located in the ``PYTHONPATH``, so that it's importable from Airflow. Start by creating a directory to store the config file. ``$AIRFLOW_HOME/config`` is recommended.
+#. Create empty files called ``$AIRFLOW_HOME/config/log_config.py`` and ``$AIRFLOW_HOME/config/__init__.py``.
+#. Copy the contents of ``airflow/config_templates/airflow_local_settings.py`` into the ``log_config.py`` file that was just created in the step above.
+#. Customize the following portions of the template:
+
+ .. code-block:: bash
+
+ # wasb buckets should start with "wasb" just to help Airflow select correct handler
+ REMOTE_BASE_LOG_FOLDER = 'wasb-<whatever you want here>'
+
+ # Rename DEFAULT_LOGGING_CONFIG to LOGGING CONFIG
+ LOGGING_CONFIG = ...
+
+
+#. Make sure a Azure Blob Storage (Wasb) connection hook has been defined in Airflow. The hook should have read and write access to the Azure Blob Storage bucket defined above in ``REMOTE_BASE_LOG_FOLDER``.
+
+#. Update ``$AIRFLOW_HOME/airflow.cfg`` to contain:
+
+ .. code-block:: bash
+
+ remote_logging = True
+ logging_config_class = log_config.LOGGING_CONFIG
+ remote_log_conn_id = <name of the Azure Blob Storage connection>
+
+#. Restart the Airflow webserver and scheduler, and trigger (or wait for) a new task execution.
+#. Verify that logs are showing up for newly executed tasks in the bucket you've defined.
+
+.. _write-logs-gcp:
+
+Writing Logs to Google Cloud Storage
+------------------------------------
+
+Follow the steps below to enable Google Cloud Storage logging.
+
+#. Airflow's logging system requires a custom .py file to be located in the ``PYTHONPATH``, so that it's importable from Airflow. Start by creating a directory to store the config file. ``$AIRFLOW_HOME/config`` is recommended.
+#. Create empty files called ``$AIRFLOW_HOME/config/log_config.py`` and ``$AIRFLOW_HOME/config/__init__.py``.
+#. Copy the contents of ``airflow/config_templates/airflow_local_settings.py`` into the ``log_config.py`` file that was just created in the step above.
+#. Customize the following portions of the template:
+
+ .. code-block:: bash
+
+ # Add this variable to the top of the file. Note the trailing slash.
+ GCS_LOG_FOLDER = 'gs://<bucket where logs should be persisted>/'
+
+ # Rename DEFAULT_LOGGING_CONFIG to LOGGING CONFIG
+ LOGGING_CONFIG = ...
+
+ # Add a GCSTaskHandler to the 'handlers' block of the LOGGING_CONFIG variable
+ 'gcs.task': {
+ 'class': 'airflow.utils.log.gcs_task_handler.GCSTaskHandler',
+ 'formatter': 'airflow.task',
+ 'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
+ 'gcs_log_folder': GCS_LOG_FOLDER,
+ 'filename_template': FILENAME_TEMPLATE,
+ },
+
+ # Update the airflow.task and airflow.tas_runner blocks to be 'gcs.task' instead of 'file.task'.
+ 'loggers': {
+ 'airflow.task': {
+ 'handlers': ['gcs.task'],
+ ...
+ },
+ 'airflow.task_runner': {
+ 'handlers': ['gcs.task'],
+ ...
+ },
+ 'airflow': {
+ 'handlers': ['console'],
+ ...
+ },
+ }
+
+#. Make sure a Google Cloud Platform connection hook has been defined in Airflow. The hook should have read and write access to the Google Cloud Storage bucket defined above in ``GCS_LOG_FOLDER``.
+
+#. Update ``$AIRFLOW_HOME/airflow.cfg`` to contain:
+
+ .. code-block:: bash
+
+ task_log_reader = gcs.task
+ logging_config_class = log_config.LOGGING_CONFIG
+ remote_log_conn_id = <name of the Google cloud platform hook>
+
+#. Restart the Airflow webserver and scheduler, and trigger (or wait for) a new task execution.
+#. Verify that logs are showing up for newly executed tasks in the bucket you've defined.
+#. Verify that the Google Cloud Storage viewer is working in the UI. Pull up a newly executed task, and verify that you see something like:
+
+ .. code-block:: bash
+
+ *** Reading remote log from gs://<bucket where logs should be persisted>/example_bash_operator/run_this_last/2017-10-03T00:00:00/16.log.
+ [2017-10-03 21:57:50,056] {cli.py:377} INFO - Running on host chrisr-00532
+ [2017-10-03 21:57:50,093] {base_task_runner.py:115} INFO - Running: ['bash', '-c', u'airflow run example_bash_operator run_this_last 2017-10-03T00:00:00 --job_id 47 --raw -sd DAGS_FOLDER/example_dags/example_bash_operator.py']
+ [2017-10-03 21:57:51,264] {base_task_runner.py:98} INFO - Subtask: [2017-10-03 21:57:51,263] {__init__.py:45} INFO - Using executor SequentialExecutor
+ [2017-10-03 21:57:51,306] {base_task_runner.py:98} INFO - Subtask: [2017-10-03 21:57:51,306] {models.py:186} INFO - Filling up the DagBag from /airflow/dags/example_dags/example_bash_operator.py
+
+Note the top line that says it's reading from the remote log file.
+
+Please be aware that if you were persisting logs to Google Cloud Storage
+using the old-style airflow.cfg configuration method, the old logs will no
+longer be visible in the Airflow UI, though they'll still exist in Google
+Cloud Storage. This is a backwards incompatbile change. If you are unhappy
+with it, you can change the ``FILENAME_TEMPLATE`` to reflect the old-style
+log filename format.
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/index.rst
----------------------------------------------------------------------
diff --git a/docs/index.rst b/docs/index.rst
index 125a1fb..4cd96ea 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -75,7 +75,7 @@ Content
start
installation
tutorial
- configuration
+ howto/index
ui
concepts
profiling
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/installation.rst
----------------------------------------------------------------------
diff --git a/docs/installation.rst b/docs/installation.rst
index 20d0c35..9e9ab3e 100644
--- a/docs/installation.rst
+++ b/docs/installation.rst
@@ -15,9 +15,9 @@ You can also install Airflow with support for extra features like ``s3`` or ``po
.. code-block:: bash
pip install "apache-airflow[s3, postgres]"
-
+
.. note:: GPL dependency
-
+
One of the dependencies of Apache Airflow by default pulls in a GPL library ('unidecode').
In case this is a concern you can force a non GPL library by issuing
``export SLUGIFY_USES_TEXT_UNIDECODE=yes`` and then proceed with the normal installation.
@@ -75,7 +75,7 @@ Here's the list of the subpackages and what they enable:
+---------------+----------------------------------------------+-------------------------------------------------+
| mysql | ``pip install apache-airflow[mysql]`` | MySQL operators and hook, support as |
| | | an Airflow backend. The version of MySQL server |
-| | | has to be 5.6.4+. The exact version upper bound |
+| | | has to be 5.6.4+. The exact version upper bound |
| | | depends on version of ``mysqlclient`` package. |
| | | For example, ``mysqlclient`` 1.3.12 can only be |
| | | used with MySQL server 5.6.4 through 5.7. |
@@ -106,12 +106,13 @@ Here's the list of the subpackages and what they enable:
Initiating Airflow Database
'''''''''''''''''''''''''''
-Airflow requires a database to be initiated before you can run tasks. If you're just
-experimenting and learning Airflow, you can stick with the default SQLite option. If
-you don't want to use SQLite, then take a look at :doc:`configuration` to setup a
-different database.
+Airflow requires a database to be initiated before you can run tasks. If
+you're just experimenting and learning Airflow, you can stick with the
+default SQLite option. If you don't want to use SQLite, then take a look at
+:doc:`howto/initialize-database` to setup a different database.
-After configuration, you'll need to initialize the database before you can run tasks:
+After configuration, you'll need to initialize the database before you can
+run tasks:
.. code-block:: bash
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/integration.rst
----------------------------------------------------------------------
diff --git a/docs/integration.rst b/docs/integration.rst
index d488737..35e0ca0 100644
--- a/docs/integration.rst
+++ b/docs/integration.rst
@@ -70,7 +70,7 @@ Azure: Microsoft Azure
----------------------
Airflow has limited support for Microsoft Azure: interfaces exist only for Azure Blob
-Storage and Azure Data Lake. Hook, Sensor and Operator for Blob Storage and
+Storage and Azure Data Lake. Hook, Sensor and Operator for Blob Storage and
Azure Data Lake Hook are in contrib section.
Azure Blob Storage
@@ -118,34 +118,7 @@ Logging
'''''''
Airflow can be configured to read and write task logs in Azure Blob Storage.
-Follow the steps below to enable Azure Blob Storage logging.
-
-#. Airflow's logging system requires a custom .py file to be located in the ``PYTHONPATH``, so that it's importable from Airflow. Start by creating a directory to store the config file. ``$AIRFLOW_HOME/config`` is recommended.
-#. Create empty files called ``$AIRFLOW_HOME/config/log_config.py`` and ``$AIRFLOW_HOME/config/__init__.py``.
-#. Copy the contents of ``airflow/config_templates/airflow_local_settings.py`` into the ``log_config.py`` file that was just created in the step above.
-#. Customize the following portions of the template:
-
- .. code-block:: bash
-
- # wasb buckets should start with "wasb" just to help Airflow select correct handler
- REMOTE_BASE_LOG_FOLDER = 'wasb-<whatever you want here>'
-
- # Rename DEFAULT_LOGGING_CONFIG to LOGGING CONFIG
- LOGGING_CONFIG = ...
-
-
-#. Make sure a Azure Blob Storage (Wasb) connection hook has been defined in Airflow. The hook should have read and write access to the Azure Blob Storage bucket defined above in ``REMOTE_BASE_LOG_FOLDER``.
-
-#. Update ``$AIRFLOW_HOME/airflow.cfg`` to contain:
-
- .. code-block:: bash
-
- remote_logging = True
- logging_config_class = log_config.LOGGING_CONFIG
- remote_log_conn_id = <name of the Azure Blob Storage connection>
-
-#. Restart the Airflow webserver and scheduler, and trigger (or wait for) a new task execution.
-#. Verify that logs are showing up for newly executed tasks in the bucket you've defined.
+See :ref:`write-logs-azure`.
Azure Data Lake
''''''''''''''''''
@@ -345,72 +318,8 @@ they can have breaking changes between minor releases.
Logging
'''''''
-Airflow can be configured to read and write task logs in Google cloud storage.
-Follow the steps below to enable Google cloud storage logging.
-
-#. Airflow's logging system requires a custom .py file to be located in the ``PYTHONPATH``, so that it's importable from Airflow. Start by creating a directory to store the config file. ``$AIRFLOW_HOME/config`` is recommended.
-#. Create empty files called ``$AIRFLOW_HOME/config/log_config.py`` and ``$AIRFLOW_HOME/config/__init__.py``.
-#. Copy the contents of ``airflow/config_templates/airflow_local_settings.py`` into the ``log_config.py`` file that was just created in the step above.
-#. Customize the following portions of the template:
-
- .. code-block:: bash
-
- # Add this variable to the top of the file. Note the trailing slash.
- GCS_LOG_FOLDER = 'gs://<bucket where logs should be persisted>/'
-
- # Rename DEFAULT_LOGGING_CONFIG to LOGGING CONFIG
- LOGGING_CONFIG = ...
-
- # Add a GCSTaskHandler to the 'handlers' block of the LOGGING_CONFIG variable
- 'gcs.task': {
- 'class': 'airflow.utils.log.gcs_task_handler.GCSTaskHandler',
- 'formatter': 'airflow.task',
- 'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
- 'gcs_log_folder': GCS_LOG_FOLDER,
- 'filename_template': FILENAME_TEMPLATE,
- },
-
- # Update the airflow.task and airflow.tas_runner blocks to be 'gcs.task' instead of 'file.task'.
- 'loggers': {
- 'airflow.task': {
- 'handlers': ['gcs.task'],
- ...
- },
- 'airflow.task_runner': {
- 'handlers': ['gcs.task'],
- ...
- },
- 'airflow': {
- 'handlers': ['console'],
- ...
- },
- }
-
-#. Make sure a Google cloud platform connection hook has been defined in Airflow. The hook should have read and write access to the Google cloud storage bucket defined above in ``GCS_LOG_FOLDER``.
-
-#. Update ``$AIRFLOW_HOME/airflow.cfg`` to contain:
-
- .. code-block:: bash
-
- task_log_reader = gcs.task
- logging_config_class = log_config.LOGGING_CONFIG
- remote_log_conn_id = <name of the Google cloud platform hook>
-
-#. Restart the Airflow webserver and scheduler, and trigger (or wait for) a new task execution.
-#. Verify that logs are showing up for newly executed tasks in the bucket you've defined.
-#. Verify that the Google cloud storage viewer is working in the UI. Pull up a newly executed task, and verify that you see something like:
-
- .. code-block:: bash
-
- *** Reading remote log from gs://<bucket where logs should be persisted>/example_bash_operator/run_this_last/2017-10-03T00:00:00/16.log.
- [2017-10-03 21:57:50,056] {cli.py:377} INFO - Running on host chrisr-00532
- [2017-10-03 21:57:50,093] {base_task_runner.py:115} INFO - Running: ['bash', '-c', u'airflow run example_bash_operator run_this_last 2017-10-03T00:00:00 --job_id 47 --raw -sd DAGS_FOLDER/example_dags/example_bash_operator.py']
- [2017-10-03 21:57:51,264] {base_task_runner.py:98} INFO - Subtask: [2017-10-03 21:57:51,263] {__init__.py:45} INFO - Using executor SequentialExecutor
- [2017-10-03 21:57:51,306] {base_task_runner.py:98} INFO - Subtask: [2017-10-03 21:57:51,306] {models.py:186} INFO - Filling up the DagBag from /airflow/dags/example_dags/example_bash_operator.py
-
-Note the top line that says it's reading from the remote log file.
-
-Please be aware that if you were persisting logs to Google cloud storage using the old-style airflow.cfg configuration method, the old logs will no longer be visible in the Airflow UI, though they'll still exist in Google cloud storage. This is a backwards incompatbile change. If you are unhappy with it, you can change the ``FILENAME_TEMPLATE`` to reflect the old-style log filename format.
+Airflow can be configured to read and write task logs in Google Cloud Storage.
+See :ref:`write-logs-gcp`.
BigQuery
''''''''
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/084bc913/docs/start.rst
----------------------------------------------------------------------
diff --git a/docs/start.rst b/docs/start.rst
index f9f09e7..a3e21f9 100644
--- a/docs/start.rst
+++ b/docs/start.rst
@@ -51,4 +51,4 @@ run the commands below.
What's Next?
''''''''''''
-From this point, you can head to the :doc:`tutorial` section for further examples or the :doc:`configuration` section if you're ready to get your hands dirty.
+From this point, you can head to the :doc:`tutorial` section for further examples or the :doc:`howto/index` section if you're ready to get your hands dirty.