You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by cr...@apache.org on 2018/01/03 17:48:06 UTC
[06/35] incubator-airflow-site git commit: 1.9.0
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/concepts.html
----------------------------------------------------------------------
diff --git a/concepts.html b/concepts.html
index a329ae8..16d995c 100644
--- a/concepts.html
+++ b/concepts.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -208,23 +213,23 @@ workflows.</p>
<h2>Core Ideas<a class="headerlink" href="#core-ideas" title="Permalink to this headline">¶</a></h2>
<div class="section" id="dags">
<h3>DAGs<a class="headerlink" href="#dags" title="Permalink to this headline">¶</a></h3>
-<p>In Airflow, a <code class="docutils literal"><span class="pre">DAG</span></code> – or a Directed Acyclic Graph – is a collection of all
+<p>In Airflow, a <code class="docutils literal"><span class="pre">DAG</span></code> – or a Directed Acyclic Graph – is a collection of all
the tasks you want to run, organized in a way that reflects their relationships
and dependencies.</p>
<p>For example, a simple DAG could consist of three tasks: A, B, and C. It could
say that A has to run successfully before B can run, but C can run anytime. It
could say that task A times out after 5 minutes, and B can be restarted up to 5
times in case it fails. It might also say that the workflow will run every night
-at 10pm, but shouldn’t start until a certain date.</p>
+at 10pm, but shouldn’t start until a certain date.</p>
<p>In this way, a DAG describes <em>how</em> you want to carry out your workflow; but
-notice that we haven’t said anything about <em>what</em> we actually want to do! A, B,
+notice that we haven’t said anything about <em>what</em> we actually want to do! A, B,
and C could be anything. Maybe A prepares data for B to analyze while C sends an
email. Or perhaps A monitors your location so B can open your garage door while
-C turns on your house lights. The important thing is that the DAG isn’t
+C turns on your house lights. The important thing is that the DAG isn’t
concerned with what its constituent tasks do; its job is to make sure that
whatever they do happens at the right time, or in the right order, or with the
right handling of any unexpected issues.</p>
-<p>DAGs are defined in standard Python files that are placed in Airflow’s
+<p>DAGs are defined in standard Python files that are placed in Airflow’s
<code class="docutils literal"><span class="pre">DAG_FOLDER</span></code>. Airflow will execute the code in each file to dynamically build
the <code class="docutils literal"><span class="pre">DAG</span></code> objects. You can have as many DAGs as you want, each describing an
arbitrary number of tasks. In general, each one should correspond to a single
@@ -245,7 +250,7 @@ scope.</p>
</div>
<p>Sometimes this can be put to good use. For example, a common pattern with
<code class="docutils literal"><span class="pre">SubDagOperator</span></code> is to define the subdag inside a function so that Airflow
-doesn’t try to load it as a standalone DAG.</p>
+doesn’t try to load it as a standalone DAG.</p>
</div>
<div class="section" id="default-arguments">
<h4>Default Arguments<a class="headerlink" href="#default-arguments" title="Permalink to this headline">¶</a></h4>
@@ -278,13 +283,13 @@ any of its operators. This makes it easy to apply a common parameter to many ope
<p>While DAGs describe <em>how</em> to run a workflow, <code class="docutils literal"><span class="pre">Operators</span></code> determine what
actually gets done.</p>
<p>An operator describes a single task in a workflow. Operators are usually (but
-not always) atomic, meaning they can stand on their own and don’t need to share
+not always) atomic, meaning they can stand on their own and don’t need to share
resources with any other operators. The DAG will make sure that operators run in
the correct certain order; other than those dependencies, operators generally
run independently. In fact, they may run on two completely different machines.</p>
<p>This is a subtle but very important point: in general, if two operators need to
share information, like a filename or small amount of data, you should consider
-combining them into a single operator. If it absolutely can’t be avoided,
+combining them into a single operator. If it absolutely can’t be avoided,
Airflow does have a feature for operator cross-communication called XCom that is
described elsewhere in this document.</p>
<p>Airflow provides operators for many common tasks, including:</p>
@@ -293,14 +298,14 @@ described elsewhere in this document.</p>
<li><code class="docutils literal"><span class="pre">PythonOperator</span></code> - calls an arbitrary Python function</li>
<li><code class="docutils literal"><span class="pre">EmailOperator</span></code> - sends an email</li>
<li><code class="docutils literal"><span class="pre">HTTPOperator</span></code> - sends an HTTP request</li>
-<li><code class="docutils literal"><span class="pre">SqlOperator</span></code> - executes a SQL command</li>
-<li><code class="docutils literal"><span class="pre">Sensor</span></code> - waits for a certain time, file, database row, S3 key, etc...</li>
+<li><code class="docutils literal"><span class="pre">MySqlOperator</span></code>, <code class="docutils literal"><span class="pre">SqliteOperator</span></code>, <code class="docutils literal"><span class="pre">PostgresOperator</span></code>, <code class="docutils literal"><span class="pre">MsSqlOperator</span></code>, <code class="docutils literal"><span class="pre">OracleOperator</span></code>, <code class="docutils literal"><span class="pre">JdbcOperator</span></code>, etc. - executes a SQL command</li>
+<li><code class="docutils literal"><span class="pre">Sensor</span></code> - waits for a certain time, file, database row, S3 key, etc…</li>
</ul>
<p>In addition to these basic building blocks, there are many more specific
operators: <code class="docutils literal"><span class="pre">DockerOperator</span></code>, <code class="docutils literal"><span class="pre">HiveOperator</span></code>, <code class="docutils literal"><span class="pre">S3FileTransferOperator</span></code>,
-<code class="docutils literal"><span class="pre">PrestoToMysqlOperator</span></code>, <code class="docutils literal"><span class="pre">SlackOperator</span></code>... you get the idea!</p>
+<code class="docutils literal"><span class="pre">PrestoToMysqlOperator</span></code>, <code class="docutils literal"><span class="pre">SlackOperator</span></code>… you get the idea!</p>
<p>The <code class="docutils literal"><span class="pre">airflow/contrib/</span></code> directory contains yet more operators built by the
-community. These operators aren’t always as complete or well-tested as those in
+community. These operators aren’t always as complete or well-tested as those in
the main distribution, but allow users to more easily add new functionality to
the platform.</p>
<p>Operators are only loaded by Airflow if they are assigned to a DAG.</p>
@@ -344,7 +349,7 @@ functionally equivalent:</p>
<p>When using the bitshift to compose operators, the relationship is set in the
direction that the bitshift operator points. For example, <code class="docutils literal"><span class="pre">op1</span> <span class="pre">>></span> <span class="pre">op2</span></code> means
that <code class="docutils literal"><span class="pre">op1</span></code> runs first and <code class="docutils literal"><span class="pre">op2</span></code> runs second. Multiple operators can be
-composed – keep in mind the chain is executed left-to-right and the rightmost
+composed – keep in mind the chain is executed left-to-right and the rightmost
object is always returned. For example:</p>
<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">op1</span> <span class="o">>></span> <span class="n">op2</span> <span class="o">>></span> <span class="n">op3</span> <span class="o"><<</span> <span class="n">op4</span>
</pre></div>
@@ -367,8 +372,7 @@ object is always returned. For example:</p>
<p>We can put this all together to build a simple pipeline:</p>
<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'my_dag'</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2016</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
<span class="p">(</span>
- <span class="n">dag</span>
- <span class="o">>></span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'dummy_1'</span><span class="p">)</span>
+ <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'dummy_1'</span><span class="p">)</span>
<span class="o">>></span> <span class="n">BashOperator</span><span class="p">(</span>
<span class="n">task_id</span><span class="o">=</span><span class="s1">'bash_1'</span><span class="p">,</span>
<span class="n">bash_command</span><span class="o">=</span><span class="s1">'echo "HELLO!"'</span><span class="p">)</span>
@@ -382,7 +386,7 @@ object is always returned. For example:</p>
</div>
<div class="section" id="tasks">
<h3>Tasks<a class="headerlink" href="#tasks" title="Permalink to this headline">¶</a></h3>
-<p>Once an operator is instantiated, it is referred to as a “task”. The
+<p>Once an operator is instantiated, it is referred to as a “task”. The
instantiation defines specific values when calling the abstract operator, and
the parameterized task becomes a node in a DAG.</p>
</div>
@@ -390,12 +394,12 @@ the parameterized task becomes a node in a DAG.</p>
<h3>Task Instances<a class="headerlink" href="#task-instances" title="Permalink to this headline">¶</a></h3>
<p>A task instance represents a specific run of a task and is characterized as the
combination of a dag, a task, and a point in time. Task instances also have an
-indicative state, which could be “running”, “success”, “failed”, “skipped”, “up
-for retry”, etc.</p>
+indicative state, which could be “running”, “success”, “failed”, “skipped”, “up
+for retry”, etc.</p>
</div>
<div class="section" id="workflows">
<h3>Workflows<a class="headerlink" href="#workflows" title="Permalink to this headline">¶</a></h3>
-<p>You’re now familiar with the core building blocks of Airflow.
+<p>You’re now familiar with the core building blocks of Airflow.
Some of the concepts may sound very similar, but the vocabulary can
be conceptualized like this:</p>
<ul class="simple">
@@ -457,8 +461,8 @@ gets prioritized accordingly.</p>
reached, runnable tasks get queued and their state will show as such in the
UI. As slots free up, queued tasks start running based on the
<code class="docutils literal"><span class="pre">priority_weight</span></code> (of the task and its descendants).</p>
-<p>Note that by default tasks aren’t assigned to any pool and their
-execution parallelism is only limited to the executor’s setting.</p>
+<p>Note that by default tasks aren’t assigned to any pool and their
+execution parallelism is only limited to the executor’s setting.</p>
</div>
<div class="section" id="connections">
<h3>Connections<a class="headerlink" href="#connections" title="Permalink to this headline">¶</a></h3>
@@ -489,7 +493,7 @@ variable to be in a URI format (e.g.
<p>When using the CeleryExecutor, the celery queues that tasks are sent to
can be specified. <code class="docutils literal"><span class="pre">queue</span></code> is an attribute of BaseOperator, so any
task can be assigned to any queue. The default queue for the environment
-is defined in the <code class="docutils literal"><span class="pre">airflow.cfg</span></code>‘s <code class="docutils literal"><span class="pre">celery</span> <span class="pre">-></span> <span class="pre">default_queue</span></code>. This defines
+is defined in the <code class="docutils literal"><span class="pre">airflow.cfg</span></code>’s <code class="docutils literal"><span class="pre">celery</span> <span class="pre">-></span> <span class="pre">default_queue</span></code>. This defines
the queue that tasks get assigned to when not specified, as well as which
queue Airflow workers listen to when started.</p>
<p>Workers can listen to one or multiple queues of tasks. When a worker is
@@ -505,16 +509,16 @@ itself because it needs a very specific environment and security rights).</p>
<div class="section" id="xcoms">
<h3>XComs<a class="headerlink" href="#xcoms" title="Permalink to this headline">¶</a></h3>
<p>XComs let tasks exchange messages, allowing more nuanced forms of control and
-shared state. The name is an abbreviation of “cross-communication”. XComs are
+shared state. The name is an abbreviation of “cross-communication”. XComs are
principally defined by a key, value, and timestamp, but also track attributes
like the task/DAG that created the XCom and when it should become visible. Any
object that can be pickled can be used as an XCom value, so users should make
sure to use objects of appropriate size.</p>
-<p>XComs can be “pushed” (sent) or “pulled” (received). When a task pushes an
+<p>XComs can be “pushed” (sent) or “pulled” (received). When a task pushes an
XCom, it makes it generally available to other tasks. Tasks can push XComs at
any time by calling the <code class="docutils literal"><span class="pre">xcom_push()</span></code> method. In addition, if a task returns
-a value (either from its Operator’s <code class="docutils literal"><span class="pre">execute()</span></code> method, or from a
-PythonOperator’s <code class="docutils literal"><span class="pre">python_callable</span></code> function), then an XCom containing that
+a value (either from its Operator’s <code class="docutils literal"><span class="pre">execute()</span></code> method, or from a
+PythonOperator’s <code class="docutils literal"><span class="pre">python_callable</span></code> function), then an XCom containing that
value is automatically pushed.</p>
<p>Tasks call <code class="docutils literal"><span class="pre">xcom_pull()</span></code> to retrieve XComs, optionally applying filters
based on criteria like <code class="docutils literal"><span class="pre">key</span></code>, source <code class="docutils literal"><span class="pre">task_ids</span></code>, and source <code class="docutils literal"><span class="pre">dag_id</span></code>. By
@@ -533,7 +537,7 @@ passed, then a correpsonding list of XCom values is returned.</p>
<span class="n">value</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'task_instance'</span><span class="p">]</span><span class="o">.</span><span class="n">xcom_pull</span><span class="p">(</span><span class="n">task_ids</span><span class="o">=</span><span class="s1">'pushing_task'</span><span class="p">)</span>
</pre></div>
</div>
-<p>It is also possible to pull XCom directly in a template, here’s an example
+<p>It is also possible to pull XCom directly in a template, here’s an example
of what this may look like:</p>
<div class="code sql highlight-default"><div class="highlight"><pre><span></span><span class="n">SELECT</span> <span class="o">*</span> <span class="n">FROM</span> <span class="p">{{</span> <span class="n">task_instance</span><span class="o">.</span><span class="n">xcom_pull</span><span class="p">(</span><span class="n">task_ids</span><span class="o">=</span><span class="s1">'foo'</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="s1">'table_name'</span><span class="p">)</span> <span class="p">}}</span>
</pre></div>
@@ -575,9 +579,9 @@ directly downstream from the BranchPythonOperator task.</p>
will invariably lead to block tasks that depend on their past successes.
<code class="docutils literal"><span class="pre">skipped</span></code> states propagates where all directly upstream tasks are
<code class="docutils literal"><span class="pre">skipped</span></code>.</p>
-<p>If you want to skip some tasks, keep in mind that you can’t have an empty
+<p>If you want to skip some tasks, keep in mind that you can’t have an empty
path, if so make a dummy task.</p>
-<p>like this, the dummy task “branch_false” is skipped</p>
+<p>like this, the dummy task “branch_false” is skipped</p>
<img alt="_images/branch_good.png" src="_images/branch_good.png" />
<p>Not like this, where the join task is skipped</p>
<img alt="_images/branch_bad.png" src="_images/branch_bad.png" />
@@ -649,11 +653,11 @@ the tasks contained within the SubDAG:</p>
<img alt="_images/subdag_zoom.png" src="_images/subdag_zoom.png" />
<p>Some other tips when using SubDAGs:</p>
<ul class="simple">
-<li>by convention, a SubDAG’s <code class="docutils literal"><span class="pre">dag_id</span></code> should be prefixed by its parent and
+<li>by convention, a SubDAG’s <code class="docutils literal"><span class="pre">dag_id</span></code> should be prefixed by its parent and
a dot. As in <code class="docutils literal"><span class="pre">parent.child</span></code></li>
<li>share arguments between the main DAG and the SubDAG by passing arguments to
the SubDAG operator (as demonstrated above)</li>
-<li>SubDAGs must have a schedule and be enabled. If the SubDAG’s schedule is
+<li>SubDAGs must have a schedule and be enabled. If the SubDAG’s schedule is
set to <code class="docutils literal"><span class="pre">None</span></code> or <code class="docutils literal"><span class="pre">@once</span></code>, the SubDAG will succeed without having done
anything</li>
<li>clearing a SubDagOperator also clears the state of the tasks within</li>
@@ -685,8 +689,8 @@ directly upstream tasks have succeeded, Airflow allows for more complex
dependency settings.</p>
<p>All operators have a <code class="docutils literal"><span class="pre">trigger_rule</span></code> argument which defines the rule by which
the generated task get triggered. The default value for <code class="docutils literal"><span class="pre">trigger_rule</span></code> is
-<code class="docutils literal"><span class="pre">all_success</span></code> and can be defined as “trigger this task when all directly
-upstream tasks have succeeded”. All other rules described here are based
+<code class="docutils literal"><span class="pre">all_success</span></code> and can be defined as “trigger this task when all directly
+upstream tasks have succeeded”. All other rules described here are based
on direct parent tasks and are values that can be passed to any operator
while creating tasks:</p>
<ul class="simple">
@@ -699,7 +703,7 @@ while creating tasks:</p>
</ul>
<p>Note that these can be used in conjunction with <code class="docutils literal"><span class="pre">depends_on_past</span></code> (boolean)
that, when set to <code class="docutils literal"><span class="pre">True</span></code>, keeps a task from getting triggered if the
-previous schedule for the task hasn’t succeeded.</p>
+previous schedule for the task hasn’t succeeded.</p>
</div>
<div class="section" id="latest-run-only">
<h3>Latest Run Only<a class="headerlink" href="#latest-run-only" title="Permalink to this headline">¶</a></h3>
@@ -772,16 +776,16 @@ and <code class="docutils literal"><span class="pre">task2</span></code> has suc
but sometimes unexpectedly.</p>
<p>Zombie tasks are characterized by the absence
of an heartbeat (emitted by the job periodically) and a <code class="docutils literal"><span class="pre">running</span></code> status
-in the database. They can occur when a worker node can’t reach the database,
+in the database. They can occur when a worker node can’t reach the database,
when Airflow processes are killed externally, or when a node gets rebooted
-for instance. Zombie killing is performed periodically by the scheduler’s
+for instance. Zombie killing is performed periodically by the scheduler’s
process.</p>
<p>Undead processes are characterized by the existence of a process and a matching
-heartbeat, but Airflow isn’t aware of this task as <code class="docutils literal"><span class="pre">running</span></code> in the database.
+heartbeat, but Airflow isn’t aware of this task as <code class="docutils literal"><span class="pre">running</span></code> in the database.
This mismatch typically occurs as the state of the database is altered,
-most likely by deleting rows in the “Task Instances” view in the UI.
+most likely by deleting rows in the “Task Instances” view in the UI.
Tasks are instructed to verify their state as part of the heartbeat routine,
-and terminate themselves upon figuring out that they are in this “undead”
+and terminate themselves upon figuring out that they are in this “undead”
state.</p>
</div>
<div class="section" id="cluster-policy">
@@ -792,10 +796,10 @@ attributes. It receives a single argument as a reference to task objects,
and is expected to alter its attributes.</p>
<p>For example, this function could apply a specific queue property when
using a specific operator, or enforce a task timeout policy, making sure
-that no tasks run for more than 48 hours. Here’s an example of what this
+that no tasks run for more than 48 hours. Here’s an example of what this
may look like inside your <code class="docutils literal"><span class="pre">airflow_settings.py</span></code>:</p>
<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">policy</span><span class="p">(</span><span class="n">task</span><span class="p">):</span>
- <span class="k">if</span> <span class="n">task</span><span class="o">.</span><span class="n">__class__</span><span class="o">.</span><span class="n">__name__</span> <span class="o">==</span> <span class="s1">'HivePartitionSensor'</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">task</span><span class="o">.</span><span class="vm">__class__</span><span class="o">.</span><span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'HivePartitionSensor'</span><span class="p">:</span>
<span class="n">task</span><span class="o">.</span><span class="n">queue</span> <span class="o">=</span> <span class="s2">"sensor_queue"</span>
<span class="k">if</span> <span class="n">task</span><span class="o">.</span><span class="n">timeout</span> <span class="o">></span> <span class="n">timedelta</span><span class="p">(</span><span class="n">hours</span><span class="o">=</span><span class="mi">48</span><span class="p">):</span>
<span class="n">task</span><span class="o">.</span><span class="n">timeout</span> <span class="o">=</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">hours</span><span class="o">=</span><span class="mi">48</span><span class="p">)</span>
@@ -804,8 +808,8 @@ may look like inside your <code class="docutils literal"><span class="pre">airfl
</div>
<div class="section" id="documentation-notes">
<h3>Documentation & Notes<a class="headerlink" href="#documentation-notes" title="Permalink to this headline">¶</a></h3>
-<p>It’s possible to add documentation or notes to your dags & task objects that
-become visible in the web interface (“Graph View” for dags, “Task Details” for
+<p>It’s possible to add documentation or notes to your dags & task objects that
+become visible in the web interface (“Graph View” for dags, “Task Details” for
tasks). There are a set of special task attributes that get rendered as rich
content if defined:</p>
<table border="1" class="docutils">
@@ -845,7 +849,7 @@ to the related tasks in Airflow.</p>
<span class="sd">"""</span>
<span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'my_dag'</span><span class="p">,</span> <span class="n">default_args</span><span class="o">=</span><span class="n">default_args</span><span class="p">)</span>
-<span class="n">dag</span><span class="o">.</span><span class="n">doc_md</span> <span class="o">=</span> <span class="n">__doc__</span>
+<span class="n">dag</span><span class="o">.</span><span class="n">doc_md</span> <span class="o">=</span> <span class="vm">__doc__</span>
<span class="n">t</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span><span class="s2">"foo"</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
<span class="n">t</span><span class="o">.</span><span class="n">doc_md</span> <span class="o">=</span> <span class="s2">"""</span><span class="se">\</span>
@@ -854,11 +858,11 @@ to the related tasks in Airflow.</p>
<span class="s2">"""</span>
</pre></div>
</div>
-<p>This content will get rendered as markdown respectively in the “Graph View” and
-“Task Details” pages.</p>
+<p>This content will get rendered as markdown respectively in the “Graph View” and
+“Task Details” pages.</p>
</div>
<div class="section" id="jinja-templating">
-<h3>Jinja Templating<a class="headerlink" href="#jinja-templating" title="Permalink to this headline">¶</a></h3>
+<span id="id1"></span><h3>Jinja Templating<a class="headerlink" href="#jinja-templating" title="Permalink to this headline">¶</a></h3>
<p>Airflow leverages the power of
<a class="reference external" href="http://jinja.pocoo.org/docs/dev/">Jinja Templating</a> and this can be a
powerful tool to use in combination with macros (see the <a class="reference internal" href="code.html#macros"><span class="std std-ref">Macros</span></a> section).</p>
@@ -876,8 +880,9 @@ to a Bash script using the <code class="docutils literal"><span class="pre">Bash
<p>Here, <code class="docutils literal"><span class="pre">{{</span> <span class="pre">ds</span> <span class="pre">}}</span></code> is a macro, and because the <code class="docutils literal"><span class="pre">env</span></code> parameter of the
<code class="docutils literal"><span class="pre">BashOperator</span></code> is templated with Jinja, the execution date will be available
as an environment variable named <code class="docutils literal"><span class="pre">EXECUTION_DATE</span></code> in your Bash script.</p>
-<p>You can use Jinja templating with every parameter that is marked as “templated”
-in the documentation.</p>
+<p>You can use Jinja templating with every parameter that is marked as “templated”
+in the documentation. Template substitution occurs just before the pre_execute
+function of your operator is called.</p>
</div>
</div>
<div class="section" id="packaged-dags">
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/configuration.html
----------------------------------------------------------------------
diff --git a/configuration.html b/configuration.html
index ea1d2e9..881716e 100644
--- a/configuration.html
+++ b/configuration.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -92,6 +97,7 @@
<li class="toctree-l2"><a class="reference internal" href="#setting-up-a-backend">Setting up a Backend</a></li>
<li class="toctree-l2"><a class="reference internal" href="#connections">Connections</a></li>
<li class="toctree-l2"><a class="reference internal" href="#scaling-out-with-celery">Scaling Out with Celery</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#scaling-out-with-dask">Scaling Out with Dask</a></li>
<li class="toctree-l2"><a class="reference internal" href="#logs">Logs</a></li>
<li class="toctree-l2"><a class="reference internal" href="#scaling-out-on-mesos-community-contributed">Scaling Out on Mesos (community contributed)</a></li>
<li class="toctree-l2"><a class="reference internal" href="#integration-with-systemd">Integration with systemd</a></li>
@@ -180,7 +186,7 @@ building a production-grade environment requires a bit more work!</p>
<div class="section" id="setting-configuration-options">
<span id="setting-options"></span><h2>Setting Configuration Options<a class="headerlink" href="#setting-configuration-options" title="Permalink to this headline">¶</a></h2>
<p>The first time you run Airflow, it will create a file called <code class="docutils literal"><span class="pre">airflow.cfg</span></code> in
-your <code class="docutils literal"><span class="pre">$AIRFLOW_HOME</span></code> directory (<code class="docutils literal"><span class="pre">~/airflow</span></code> by default). This file contains Airflow’s configuration and you
+your <code class="docutils literal"><span class="pre">$AIRFLOW_HOME</span></code> directory (<code class="docutils literal"><span class="pre">~/airflow</span></code> by default). This file contains Airflow’s configuration and you
can edit it to change any of the settings. You can also set options with environment variables by using this format:
<code class="docutils literal"><span class="pre">$AIRFLOW__{SECTION}__{KEY}</span></code> (note the double underscores).</p>
<p>For example, the
@@ -222,10 +228,10 @@ specific schema in the Postgres connection URI, you may
want to set a default schema for your role with a
command similar to <code class="docutils literal"><span class="pre">ALTER</span> <span class="pre">ROLE</span> <span class="pre">username</span> <span class="pre">SET</span> <span class="pre">search_path</span> <span class="pre">=</span> <span class="pre">airflow,</span> <span class="pre">foobar;</span></code></p>
</div>
-<p>Once you’ve setup your database to host Airflow, you’ll need to alter the
+<p>Once you’ve setup your database to host Airflow, you’ll need to alter the
SqlAlchemy connection string located in your configuration file
-<code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code>. You should then also change the “executor”
-setting to use “LocalExecutor”, an executor that can parallelize task
+<code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code>. You should then also change the “executor”
+setting to use “LocalExecutor”, an executor that can parallelize task
instances locally.</p>
<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># initialize the database</span>
airflow initdb
@@ -237,12 +243,35 @@ airflow initdb
<p>Airflow needs to know how to connect to your environment. Information
such as hostname, port, login and passwords to other systems and services is
handled in the <code class="docutils literal"><span class="pre">Admin->Connection</span></code> section of the UI. The pipeline code you
-will author will reference the ‘conn_id’ of the Connection objects.</p>
+will author will reference the ‘conn_id’ of the Connection objects.</p>
<img alt="_images/connections.png" src="_images/connections.png" />
<p>By default, Airflow will save the passwords for the connection in plain text
within the metadata database. The <code class="docutils literal"><span class="pre">crypto</span></code> package is highly recommended
during installation. The <code class="docutils literal"><span class="pre">crypto</span></code> package does require that your operating
system have libffi-dev installed.</p>
+<p>If <code class="docutils literal"><span class="pre">crypto</span></code> package was not installed initially, you can still enable encryption for
+connections by following steps below:</p>
+<ol class="arabic simple">
+<li>Install crypto package <code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">apache-airflow[crypto]</span></code></li>
+<li>Generate fernet_key, using this code snippet below. fernet_key must be a base64-encoded 32-byte key.</li>
+</ol>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">cryptography.fernet</span> <span class="k">import</span> <span class="n">Fernet</span>
+<span class="n">fernet_key</span><span class="o">=</span> <span class="n">Fernet</span><span class="o">.</span><span class="n">generate_key</span><span class="p">()</span>
+<span class="nb">print</span><span class="p">(</span><span class="n">fernet_key</span><span class="p">)</span> <span class="c1"># your fernet_key, keep it in secured place!</span>
+</pre></div>
+</div>
+<p>3. Replace <code class="docutils literal"><span class="pre">airflow.cfg</span></code> fernet_key value with the one from step 2.
+Alternatively, you can store your fernet_key in OS environment variable. You
+do not need to change <code class="docutils literal"><span class="pre">airflow.cfg</span></code> in this case as AirFlow will use environment
+variable over the value in <code class="docutils literal"><span class="pre">airflow.cfg</span></code>:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># Note the double underscores</span>
+EXPORT <span class="nv">AIRFLOW__CORE__FERNET_KEY</span> <span class="o">=</span> your_fernet_key
+</pre></div>
+</div>
+<ol class="arabic simple" start="4">
+<li>Restart AirFlow webserver.</li>
+<li>For existing connections (the ones that you had defined before installing <code class="docutils literal"><span class="pre">airflow[crypto]</span></code> and creating a Fernet key), you need to open each connection in the connection admin UI, re-type the password, and save it.</li>
+</ol>
<p>Connections in Airflow pipelines can be created using environment variables.
The environment variable needs to have a prefix of <code class="docutils literal"><span class="pre">AIRFLOW_CONN_</span></code> for
Airflow with the value in a URI format to use the connection properly. Please
@@ -252,7 +281,7 @@ variables and connections.</p>
<div class="section" id="scaling-out-with-celery">
<h2>Scaling Out with Celery<a class="headerlink" href="#scaling-out-with-celery" title="Permalink to this headline">¶</a></h2>
<p><code class="docutils literal"><span class="pre">CeleryExecutor</span></code> is one of the ways you can scale out the number of workers. For this
-to work, you need to setup a Celery backend (<strong>RabbitMQ</strong>, <strong>Redis</strong>, ...) and
+to work, you need to setup a Celery backend (<strong>RabbitMQ</strong>, <strong>Redis</strong>, …) and
change your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> to point the executor parameter to
<code class="docutils literal"><span class="pre">CeleryExecutor</span></code> and provide the related Celery settings.</p>
<p>For more information about setting up a Celery broker, refer to the
@@ -280,10 +309,37 @@ subcommand</p>
</div>
<p>Your worker should start picking up tasks as soon as they get fired in
its direction.</p>
-<p>Note that you can also run “Celery Flower”, a web UI built on top of Celery,
+<p>Note that you can also run “Celery Flower”, a web UI built on top of Celery,
to monitor your workers. You can use the shortcut command <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">flower</span></code>
to start a Flower web server.</p>
</div>
+<div class="section" id="scaling-out-with-dask">
+<h2>Scaling Out with Dask<a class="headerlink" href="#scaling-out-with-dask" title="Permalink to this headline">¶</a></h2>
+<p><code class="docutils literal"><span class="pre">DaskExecutor</span></code> allows you to run Airflow tasks in a Dask Distributed cluster.</p>
+<p>Dask clusters can be run on a single machine or on remote networks. For complete
+details, consult the <a class="reference external" href="https://distributed.readthedocs.io/">Distributed documentation</a>.</p>
+<p>To create a cluster, first start a Scheduler:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># default settings for a local cluster</span>
+<span class="nv">DASK_HOST</span><span class="o">=</span><span class="m">127</span>.0.0.1
+<span class="nv">DASK_PORT</span><span class="o">=</span><span class="m">8786</span>
+
+dask-scheduler --host <span class="nv">$DASK_HOST</span> --port <span class="nv">$DASK_PORT</span>
+</pre></div>
+</div>
+<p>Next start at least one Worker on any machine that can connect to the host:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span>dask-worker <span class="nv">$DASK_HOST</span>:<span class="nv">$DASK_PORT</span>
+</pre></div>
+</div>
+<p>Edit your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> to set your executor to <code class="docutils literal"><span class="pre">DaskExecutor</span></code> and provide
+the Dask Scheduler address in the <code class="docutils literal"><span class="pre">[dask]</span></code> section.</p>
+<p>Please note:</p>
+<ul class="simple">
+<li>Each Dask worker must be able to import Airflow and any dependencies you
+require.</li>
+<li>Dask does not support queues. If an Airflow task was created with a queue, a
+warning will be raised but the task will be submitted to the cluster.</li>
+</ul>
+</div>
<div class="section" id="logs">
<h2>Logs<a class="headerlink" href="#logs" title="Permalink to this headline">¶</a></h2>
<p>Users can specify a logs folder in <code class="docutils literal"><span class="pre">airflow.cfg</span></code>. By default, it is in
@@ -302,13 +358,14 @@ To enable this feature, <code class="docutils literal"><span class="pre">airflow
<span class="nv">encrypt_s3_logs</span> <span class="o">=</span> False
</pre></div>
</div>
-<p>Remote logging uses an existing Airflow connection to read/write logs. If you don’t
+<p>Remote logging uses an existing Airflow connection to read/write logs. If you don’t
have a connection properly setup, this will fail. In the above example, Airflow will
try to use <code class="docutils literal"><span class="pre">S3Hook('MyS3Conn')</span></code>.</p>
<p>In the Airflow Web UI, local logs take precedance over remote logs. If local logs
can not be found or accessed, the remote logs will be displayed. Note that logs
are only sent to remote storage once a task completes (including failure). In other
-words, remote logs for running tasks are unavailable.</p>
+words, remote logs for running tasks are unavailable. Logs are stored in the log
+folder as <code class="docutils literal"><span class="pre">{dag_id}/{task_id}/{execution_date}/{try_number}.log</span></code>.</p>
</div>
<div class="section" id="scaling-out-on-mesos-community-contributed">
<h2>Scaling Out on Mesos (community contributed)<a class="headerlink" href="#scaling-out-on-mesos-community-contributed" title="Permalink to this headline">¶</a></h2>
@@ -317,7 +374,7 @@ For this to work, you need a running mesos cluster and you must perform the foll
steps -</p>
<ol class="arabic simple">
<li>Install airflow on a machine where web server and scheduler will run,
-let’s refer to this as the “Airflow server”.</li>
+let’s refer to this as the “Airflow server”.</li>
<li>On the Airflow server, install mesos python eggs from <a class="reference external" href="http://open.mesosphere.com/downloads/mesos/">mesos downloads</a>.</li>
<li>On the Airflow server, use a database (such as mysql) which can be accessed from mesos
slave machines and add configuration in <code class="docutils literal"><span class="pre">airflow.cfg</span></code>.</li>
@@ -374,7 +431,7 @@ integrated with upstart</p>
</div>
<div class="section" id="test-mode">
<h2>Test Mode<a class="headerlink" href="#test-mode" title="Permalink to this headline">¶</a></h2>
-<p>Airflow has a fixed set of “test mode” configuration options. You can load these
+<p>Airflow has a fixed set of “test mode” configuration options. You can load these
at any time by calling <code class="docutils literal"><span class="pre">airflow.configuration.load_test_config()</span></code> (note this
operation is not reversible!). However, some options (like the DAG_FOLDER) are
loaded before you have a chance to call load_test_config(). In order to eagerly load
@@ -383,7 +440,7 @@ the test configuration, set test_mode in airflow.cfg:</p>
<span class="nv">unit_test_mode</span> <span class="o">=</span> True
</pre></div>
</div>
-<p>Due to Airflow’s automatic environment variable expansion (see <a class="reference internal" href="#setting-options"><span class="std std-ref">Setting Configuration Options</span></a>),
+<p>Due to Airflow’s automatic environment variable expansion (see <a class="reference internal" href="#setting-options"><span class="std std-ref">Setting Configuration Options</span></a>),
you can also set the env var <code class="docutils literal"><span class="pre">AIRFLOW__CORE__UNIT_TEST_MODE</span></code> to temporarily overwrite
airflow.cfg.</p>
</div>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/28a3eb60/faq.html
----------------------------------------------------------------------
diff --git a/faq.html b/faq.html
index 7b6c8ab..f223045 100644
--- a/faq.html
+++ b/faq.html
@@ -13,6 +13,8 @@
+
+
@@ -81,7 +83,10 @@
- <ul class="current">
+
+
+
+ <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
@@ -98,10 +103,10 @@
<li class="toctree-l1"><a class="reference internal" href="api.html">Experimental Rest API</a></li>
<li class="toctree-l1"><a class="reference internal" href="integration.html">Integration</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">FAQ</a><ul>
-<li class="toctree-l2"><a class="reference internal" href="#why-isn-t-my-task-getting-scheduled">Why isn’t my task getting scheduled?</a></li>
-<li class="toctree-l2"><a class="reference internal" href="#how-do-i-trigger-tasks-based-on-another-task-s-failure">How do I trigger tasks based on another task’s failure?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#why-isn-t-my-task-getting-scheduled">Why isn’t my task getting scheduled?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#how-do-i-trigger-tasks-based-on-another-task-s-failure">How do I trigger tasks based on another task’s failure?</a></li>
<li class="toctree-l2"><a class="reference internal" href="#why-are-connection-passwords-still-not-encrypted-in-the-metadata-db-after-i-installed-airflow-crypto">Why are connection passwords still not encrypted in the metadata db after I installed airflow[crypto]?</a></li>
-<li class="toctree-l2"><a class="reference internal" href="#what-s-the-deal-with-start-date">What’s the deal with <code class="docutils literal"><span class="pre">start_date</span></code>?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#what-s-the-deal-with-start-date">What’s the deal with <code class="docutils literal"><span class="pre">start_date</span></code>?</a></li>
<li class="toctree-l2"><a class="reference internal" href="#how-can-i-create-dags-dynamically">How can I create DAGs dynamically?</a></li>
<li class="toctree-l2"><a class="reference internal" href="#what-are-all-the-airflow-run-commands-in-my-process-list">What are all the <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span></code> commands in my process list?</a></li>
</ul>
@@ -173,11 +178,11 @@
<div class="section" id="faq">
<h1>FAQ<a class="headerlink" href="#faq" title="Permalink to this headline">¶</a></h1>
<div class="section" id="why-isn-t-my-task-getting-scheduled">
-<h2>Why isn’t my task getting scheduled?<a class="headerlink" href="#why-isn-t-my-task-getting-scheduled" title="Permalink to this headline">¶</a></h2>
+<h2>Why isn’t my task getting scheduled?<a class="headerlink" href="#why-isn-t-my-task-getting-scheduled" title="Permalink to this headline">¶</a></h2>
<p>There are very many reasons why your task might not be getting scheduled.
Here are some of the common causes:</p>
<ul class="simple">
-<li>Does your script “compile”, can the Airflow engine parse it and find your
+<li>Does your script “compile”, can the Airflow engine parse it and find your
DAG object. To test this, you can run <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">list_dags</span></code> and
confirm that your DAG shows up in the list. You can also run
<code class="docutils literal"><span class="pre">airflow</span> <span class="pre">list_tasks</span> <span class="pre">foo_dag_id</span> <span class="pre">--tree</span></code> and confirm that your task
@@ -189,9 +194,9 @@ task soon after the <code class="docutils literal"><span class="pre">start_date<
<li>Is your <code class="docutils literal"><span class="pre">schedule_interval</span></code> set properly? The default <code class="docutils literal"><span class="pre">schedule_interval</span></code>
is one day (<code class="docutils literal"><span class="pre">datetime.timedelta(1)</span></code>). You must specify a different <code class="docutils literal"><span class="pre">schedule_interval</span></code>
directly to the DAG object you instantiate, not as a <code class="docutils literal"><span class="pre">default_param</span></code>, as task instances
-do not override their parent DAG’s <code class="docutils literal"><span class="pre">schedule_interval</span></code>.</li>
+do not override their parent DAG’s <code class="docutils literal"><span class="pre">schedule_interval</span></code>.</li>
<li>Is your <code class="docutils literal"><span class="pre">start_date</span></code> beyond where you can see it in the UI? If you
-set your it to some time say 3 months ago, you won’t be able to see
+set your <code class="docutils literal"><span class="pre">start_date</span></code> to some time say 3 months ago, you won’t be able to see
it in the main view in the UI, but you should be able to see it in the
<code class="docutils literal"><span class="pre">Menu</span> <span class="pre">-></span> <span class="pre">Browse</span> <span class="pre">->Task</span> <span class="pre">Instances</span></code>.</li>
<li>Are the dependencies for the task met. The task instances directly
@@ -203,7 +208,7 @@ what it means.
You can view how these properties are set from the <code class="docutils literal"><span class="pre">Task</span> <span class="pre">Instance</span> <span class="pre">Details</span></code>
page for your task.</li>
<li>Are the DagRuns you need created and active? A DagRun represents a specific
-execution of an entire DAG and has a state (running, success, failed, ...).
+execution of an entire DAG and has a state (running, success, failed, …).
The scheduler creates new DagRun as it moves forward, but never goes back
in time to create new ones. The scheduler only evaluates <code class="docutils literal"><span class="pre">running</span></code> DagRuns
to see what task instances it can trigger. Note that clearing tasks
@@ -220,19 +225,17 @@ how many <code class="docutils literal"><span class="pre">running</span></code>
sure you fully understand how it proceeds.</p>
</div>
<div class="section" id="how-do-i-trigger-tasks-based-on-another-task-s-failure">
-<h2>How do I trigger tasks based on another task’s failure?<a class="headerlink" href="#how-do-i-trigger-tasks-based-on-another-task-s-failure" title="Permalink to this headline">¶</a></h2>
+<h2>How do I trigger tasks based on another task’s failure?<a class="headerlink" href="#how-do-i-trigger-tasks-based-on-another-task-s-failure" title="Permalink to this headline">¶</a></h2>
<p>Check out the <code class="docutils literal"><span class="pre">Trigger</span> <span class="pre">Rule</span></code> section in the Concepts section of the
documentation</p>
</div>
<div class="section" id="why-are-connection-passwords-still-not-encrypted-in-the-metadata-db-after-i-installed-airflow-crypto">
<h2>Why are connection passwords still not encrypted in the metadata db after I installed airflow[crypto]?<a class="headerlink" href="#why-are-connection-passwords-still-not-encrypted-in-the-metadata-db-after-i-installed-airflow-crypto" title="Permalink to this headline">¶</a></h2>
-<ul class="simple">
-<li>Verify that the <code class="docutils literal"><span class="pre">fernet_key</span></code> defined in <code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code> is a valid Fernet key. It must be a base64-encoded 32-byte key. You need to restart the webserver after you update the key</li>
-<li>For existing connections (the ones that you had defined before installing <code class="docutils literal"><span class="pre">airflow[crypto]</span></code> and creating a Fernet key), you need to open each connection in the connection admin UI, re-type the password, and save it</li>
-</ul>
+<p>Check out the <code class="docutils literal"><span class="pre">Connections</span></code> section in the Configuration section of the
+documentation</p>
</div>
<div class="section" id="what-s-the-deal-with-start-date">
-<h2>What’s the deal with <code class="docutils literal"><span class="pre">start_date</span></code>?<a class="headerlink" href="#what-s-the-deal-with-start-date" title="Permalink to this headline">¶</a></h2>
+<h2>What’s the deal with <code class="docutils literal"><span class="pre">start_date</span></code>?<a class="headerlink" href="#what-s-the-deal-with-start-date" title="Permalink to this headline">¶</a></h2>
<p><code class="docutils literal"><span class="pre">start_date</span></code> is partly legacy from the pre-DagRun era, but it is still
relevant in many ways. When creating a new DAG, you probably want to set
a global <code class="docutils literal"><span class="pre">start_date</span></code> for your tasks using <code class="docutils literal"><span class="pre">default_args</span></code>. The first
@@ -241,7 +244,7 @@ task. From that point on, the scheduler creates new DagRuns based on
your <code class="docutils literal"><span class="pre">schedule_interval</span></code> and the corresponding task instances run as your
dependencies are met. When introducing new tasks to your DAG, you need to
pay special attention to <code class="docutils literal"><span class="pre">start_date</span></code>, and may want to reactivate
-inactive DagRuns to get the new task to get onboarded properly.</p>
+inactive DagRuns to get the new task onboarded properly.</p>
<p>We recommend against using dynamic values as <code class="docutils literal"><span class="pre">start_date</span></code>, especially
<code class="docutils literal"><span class="pre">datetime.now()</span></code> as it can be quite confusing. The task is triggered
once the period closes, and in theory an <code class="docutils literal"><span class="pre">@hourly</span></code> DAG would never get to
@@ -257,16 +260,16 @@ the execution of tasks within the schedule interval.
While <code class="docutils literal"><span class="pre">schedule_interval</span></code> does allow specifying a <code class="docutils literal"><span class="pre">datetime.timedelta</span></code>
object, we recommend using the macros or cron expressions instead, as
it enforces this idea of rounded schedules.</p>
-<p>When using <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> it’s important to pay special attention
+<p>When using <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> it’s important to pay special attention
to <code class="docutils literal"><span class="pre">start_date</span></code> as the past dependency is not enforced only on the specific
-schedule of the <code class="docutils literal"><span class="pre">start_date</span></code> specified for the task. It’ also
+schedule of the <code class="docutils literal"><span class="pre">start_date</span></code> specified for the task. It’s also
important to watch DagRun activity status in time when introducing
new <code class="docutils literal"><span class="pre">depends_on_past=True</span></code>, unless you are planning on running a backfill
for the new task(s).</p>
<p>Also important to note is that the tasks <code class="docutils literal"><span class="pre">start_date</span></code>, in the context of a
-backfill CLI command, get overridden by the backfill’s command <code class="docutils literal"><span class="pre">start_date</span></code>.
+backfill CLI command, get overridden by the backfill’s command <code class="docutils literal"><span class="pre">start_date</span></code>.
This allows for a backfill on tasks that have <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> to
-actually start, if it wasn’t the case, the backfill just wouldn’t start.</p>
+actually start, if that wasn’t the case, the backfill just wouldn’t start.</p>
</div>
<div class="section" id="how-can-i-create-dags-dynamically">
<h2>How can I create DAGs dynamically?<a class="headerlink" href="#how-can-i-create-dags-dynamically" title="Permalink to this headline">¶</a></h2>
@@ -295,7 +298,7 @@ LocalExecutor, that translates into running it in a subprocess pool.</li>
command (described below) as a subprocess and is in charge of
emitting heartbeats, listening for external kill signals
and ensures some cleanup takes place if the subprocess fails</li>
-<li>Raw <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span> <span class="pre">--raw</span></code> runs the actual operator’s execute method and
+<li>Raw <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span> <span class="pre">--raw</span></code> runs the actual operator’s execute method and
performs the actual work</li>
</ul>
</div>