You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ma...@apache.org on 2017/03/06 17:10:10 UTC
[05/22] incubator-airflow-site git commit: Latest docs version as of
1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/concepts.html
----------------------------------------------------------------------
diff --git a/concepts.html b/concepts.html
index 15b2b4d..1ed51a5 100644
--- a/concepts.html
+++ b/concepts.html
@@ -30,6 +30,9 @@
+ <link rel="index" title="Index"
+ href="genindex.html"/>
+ <link rel="search" title="Search" href="search.html"/>
<link rel="top" title="Airflow Documentation" href="index.html"/>
<link rel="next" title="Data Profiling" href="profiling.html"/>
<link rel="prev" title="UI / Screenshots" href="ui.html"/>
@@ -41,6 +44,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -114,6 +118,7 @@
<li class="toctree-l3"><a class="reference internal" href="#subdags">SubDAGs</a></li>
<li class="toctree-l3"><a class="reference internal" href="#slas">SLAs</a></li>
<li class="toctree-l3"><a class="reference internal" href="#trigger-rules">Trigger Rules</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#latest-run-only">Latest Run Only</a></li>
<li class="toctree-l3"><a class="reference internal" href="#zombies-undeads">Zombies & Undeads</a></li>
<li class="toctree-l3"><a class="reference internal" href="#cluster-policy">Cluster Policy</a></li>
<li class="toctree-l3"><a class="reference internal" href="#documentation-notes">Documentation & Notes</a></li>
@@ -128,6 +133,8 @@
<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
</ul>
@@ -142,8 +149,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+
</nav>
@@ -152,23 +161,40 @@
<div class="rst-content">
-
+
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="index.html">Docs</a> »</li>
-
- <li>Concepts</li>
+
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>Concepts</li>
+
+
<li class="wy-breadcrumbs-aside">
-
- <a href="_sources/concepts.txt" rel="nofollow"> View page source</a>
+
+ <a href="_sources/concepts.rst.txt" rel="nofollow"> View page source</a>
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -341,8 +367,7 @@ object is always returned. For example:</p>
<p>We can put this all together to build a simple pipeline:</p>
<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'my_dag'</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2016</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
<span class="p">(</span>
- <span class="n">dag</span>
- <span class="o">>></span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'dummy_1'</span><span class="p">)</span>
+ <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'dummy_1'</span><span class="p">)</span>
<span class="o">>></span> <span class="n">BashOperator</span><span class="p">(</span>
<span class="n">task_id</span><span class="o">=</span><span class="s1">'bash_1'</span><span class="p">,</span>
<span class="n">bash_command</span><span class="o">=</span><span class="s1">'echo "HELLO!"'</span><span class="p">)</span>
@@ -452,10 +477,11 @@ variables from the operating system. The environment variable needs to be
prefixed with <code class="docutils literal"><span class="pre">AIRFLOW_CONN_</span></code> to be considered a connection. When
referencing the connection in the Airflow pipeline, the <code class="docutils literal"><span class="pre">conn_id</span></code> should
be the name of the variable without the prefix. For example, if the <code class="docutils literal"><span class="pre">conn_id</span></code>
-is named <code class="docutils literal"><span class="pre">POSTGRES_MASTER</span></code> the environment variable should be named
-<code class="docutils literal"><span class="pre">AIRFLOW_CONN_POSTGRES_MASTER</span></code>. Airflow assumes the value returned
-from the environment variable to be in a URI format
-(e.g. <code class="docutils literal"><span class="pre">postgres://user:password@localhost:5432/master</span></code>).</p>
+is named <code class="docutils literal"><span class="pre">postgres_master</span></code> the environment variable should be named
+<code class="docutils literal"><span class="pre">AIRFLOW_CONN_POSTGRES_MASTER</span></code> (note that the environment variable must be
+all uppercase). Airflow assumes the value returned from the environment
+variable to be in a URI format (e.g.
+<code class="docutils literal"><span class="pre">postgres://user:password@localhost:5432/master</span></code> or <code class="docutils literal"><span class="pre">s3://accesskey:secretkey@S3</span></code>).</p>
</div>
<div class="section" id="queues">
<h3>Queues<a class="headerlink" href="#queues" title="Permalink to this headline">�</a></h3>
@@ -674,6 +700,71 @@ while creating tasks:</p>
that, when set to <code class="docutils literal"><span class="pre">True</span></code>, keeps a task from getting triggered if the
previous schedule for the task hasn’t succeeded.</p>
</div>
+<div class="section" id="latest-run-only">
+<h3>Latest Run Only<a class="headerlink" href="#latest-run-only" title="Permalink to this headline">�</a></h3>
+<p>Standard workflow behavior involves running a series of tasks for a
+particular date/time range. Some workflows, however, perform tasks that
+are independent of run time but need to be run on a schedule, much like a
+standard cron job. In these cases, backfills or running jobs missed during
+a pause just wastes CPU cycles.</p>
+<p>For situations like this, you can use the <code class="docutils literal"><span class="pre">LatestOnlyOperator</span></code> to skip
+tasks that are not being run during the most recent scheduled run for a
+DAG. The <code class="docutils literal"><span class="pre">LatestOnlyOperator</span></code> skips all immediate downstream tasks, and
+itself, if the time right now is not between its <code class="docutils literal"><span class="pre">execution_time</span></code> and the
+next scheduled <code class="docutils literal"><span class="pre">execution_time</span></code>.</p>
+<p>One must be aware of the interaction between skipped tasks and trigger
+rules. Skipped tasks will cascade through trigger rules <code class="docutils literal"><span class="pre">all_success</span></code>
+and <code class="docutils literal"><span class="pre">all_failed</span></code> but not <code class="docutils literal"><span class="pre">all_done</span></code>, <code class="docutils literal"><span class="pre">one_failed</span></code>, <code class="docutils literal"><span class="pre">one_success</span></code>,
+and <code class="docutils literal"><span class="pre">dummy</span></code>. If you would like to use the <code class="docutils literal"><span class="pre">LatestOnlyOperator</span></code> with
+trigger rules that do not cascade skips, you will need to ensure that the
+<code class="docutils literal"><span class="pre">LatestOnlyOperator</span></code> is <strong>directly</strong> upstream of the task you would like
+to skip.</p>
+<p>It is possible, through use of trigger rules to mix tasks that should run
+in the typical date/time dependent mode and those using the
+<code class="docutils literal"><span class="pre">LatestOnlyOperator</span></code>.</p>
+<p>For example, consider the following dag:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="c1">#dags/latest_only_with_trigger.py</span>
+<span class="kn">import</span> <span class="nn">datetime</span> <span class="k">as</span> <span class="nn">dt</span>
+
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">DAG</span>
+<span class="kn">from</span> <span class="nn">airflow.operators.dummy_operator</span> <span class="k">import</span> <span class="n">DummyOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.operators.latest_only_operator</span> <span class="k">import</span> <span class="n">LatestOnlyOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.trigger_rule</span> <span class="k">import</span> <span class="n">TriggerRule</span>
+
+
+<span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span>
+ <span class="n">dag_id</span><span class="o">=</span><span class="s1">'latest_only_with_trigger'</span><span class="p">,</span>
+ <span class="n">schedule_interval</span><span class="o">=</span><span class="n">dt</span><span class="o">.</span><span class="n">timedelta</span><span class="p">(</span><span class="n">hours</span><span class="o">=</span><span class="mi">4</span><span class="p">),</span>
+ <span class="n">start_date</span><span class="o">=</span><span class="n">dt</span><span class="o">.</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2016</span><span class="p">,</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">20</span><span class="p">),</span>
+<span class="p">)</span>
+
+<span class="n">latest_only</span> <span class="o">=</span> <span class="n">LatestOnlyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'latest_only'</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+
+<span class="n">task1</span> <span class="o">=</span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'task1'</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+<span class="n">task1</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">latest_only</span><span class="p">)</span>
+
+<span class="n">task2</span> <span class="o">=</span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'task2'</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+
+<span class="n">task3</span> <span class="o">=</span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'task3'</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+<span class="n">task3</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">([</span><span class="n">task1</span><span class="p">,</span> <span class="n">task2</span><span class="p">])</span>
+
+<span class="n">task4</span> <span class="o">=</span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'task4'</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">,</span>
+ <span class="n">trigger_rule</span><span class="o">=</span><span class="n">TriggerRule</span><span class="o">.</span><span class="n">ALL_DONE</span><span class="p">)</span>
+<span class="n">task4</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">([</span><span class="n">task1</span><span class="p">,</span> <span class="n">task2</span><span class="p">])</span>
+</pre></div>
+</div>
+<p>In the case of this dag, the <code class="docutils literal"><span class="pre">latest_only</span></code> task will show up as skipped
+for all runs except the latest run. <code class="docutils literal"><span class="pre">task1</span></code> is directly downstream of
+<code class="docutils literal"><span class="pre">latest_only</span></code> and will also skip for all runs except the latest.
+<code class="docutils literal"><span class="pre">task2</span></code> is entirely independent of <code class="docutils literal"><span class="pre">latest_only</span></code> and will run in all
+scheduled periods. <code class="docutils literal"><span class="pre">task3</span></code> is downstream of <code class="docutils literal"><span class="pre">task1</span></code> and <code class="docutils literal"><span class="pre">task2</span></code> and
+because of the default <code class="docutils literal"><span class="pre">trigger_rule</span></code> being <code class="docutils literal"><span class="pre">all_success</span></code> will receive
+a cascaded skip from <code class="docutils literal"><span class="pre">task1</span></code>. <code class="docutils literal"><span class="pre">task4</span></code> is downstream of <code class="docutils literal"><span class="pre">task1</span></code> and
+<code class="docutils literal"><span class="pre">task2</span></code> but since its <code class="docutils literal"><span class="pre">trigger_rule</span></code> is set to <code class="docutils literal"><span class="pre">all_done</span></code> it will
+trigger as soon as <code class="docutils literal"><span class="pre">task1</span></code> has been skipped (a valid completion state)
+and <code class="docutils literal"><span class="pre">task2</span></code> has succeeded.</p>
+<img alt="_images/latest_only_with_trigger.png" src="_images/latest_only_with_trigger.png" />
+</div>
<div class="section" id="zombies-undeads">
<h3>Zombies & Undeads<a class="headerlink" href="#zombies-undeads" title="Permalink to this headline">�</a></h3>
<p>Task instances die all the time, usually as part of their normal life cycle,
@@ -842,15 +933,18 @@ pure python modules can be packaged.</p>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
- <a href="profiling.html" class="btn btn-neutral float-right" title="Data Profiling" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+ <a href="profiling.html" class="btn btn-neutral float-right" title="Data Profiling" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
- <a href="ui.html" class="btn btn-neutral" title="UI / Screenshots" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+ <a href="ui.html" class="btn btn-neutral" title="UI / Screenshots" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
</div>
@@ -883,7 +977,8 @@ pure python modules can be packaged.</p>
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/configuration.html
----------------------------------------------------------------------
diff --git a/configuration.html b/configuration.html
index ff5bd46..dc4577c 100644
--- a/configuration.html
+++ b/configuration.html
@@ -30,6 +30,9 @@
+ <link rel="index" title="Index"
+ href="genindex.html"/>
+ <link rel="search" title="Search" href="search.html"/>
<link rel="top" title="Airflow Documentation" href="index.html"/>
<link rel="next" title="UI / Screenshots" href="ui.html"/>
<link rel="prev" title="Tutorial" href="tutorial.html"/>
@@ -41,6 +44,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -88,10 +92,12 @@
<li class="toctree-l2"><a class="reference internal" href="#setting-up-a-backend">Setting up a Backend</a></li>
<li class="toctree-l2"><a class="reference internal" href="#connections">Connections</a></li>
<li class="toctree-l2"><a class="reference internal" href="#scaling-out-with-celery">Scaling Out with Celery</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#scaling-out-with-dask">Scaling Out with Dask</a></li>
<li class="toctree-l2"><a class="reference internal" href="#logs">Logs</a></li>
<li class="toctree-l2"><a class="reference internal" href="#scaling-out-on-mesos-community-contributed">Scaling Out on Mesos (community contributed)</a></li>
<li class="toctree-l2"><a class="reference internal" href="#integration-with-systemd">Integration with systemd</a></li>
<li class="toctree-l2"><a class="reference internal" href="#integration-with-upstart">Integration with upstart</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#test-mode">Test Mode</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
@@ -101,6 +107,8 @@
<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="integration.html">Integration</a></li>
<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
</ul>
@@ -115,8 +123,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+
</nav>
@@ -125,23 +135,40 @@
<div class="rst-content">
-
+
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="index.html">Docs</a> »</li>
-
- <li>Configuration</li>
+
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>Configuration</li>
+
+
<li class="wy-breadcrumbs-aside">
-
- <a href="_sources/configuration.txt" rel="nofollow"> View page source</a>
+
+ <a href="_sources/configuration.rst.txt" rel="nofollow"> View page source</a>
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -152,7 +179,7 @@
<p>Setting up the sandbox in the <a class="reference internal" href="start.html"><span class="doc">Quick Start</span></a> section was easy;
building a production-grade environment requires a bit more work!</p>
<div class="section" id="setting-configuration-options">
-<h2>Setting Configuration Options<a class="headerlink" href="#setting-configuration-options" title="Permalink to this headline">�</a></h2>
+<span id="setting-options"></span><h2>Setting Configuration Options<a class="headerlink" href="#setting-configuration-options" title="Permalink to this headline">�</a></h2>
<p>The first time you run Airflow, it will create a file called <code class="docutils literal"><span class="pre">airflow.cfg</span></code> in
your <code class="docutils literal"><span class="pre">$AIRFLOW_HOME</span></code> directory (<code class="docutils literal"><span class="pre">~/airflow</span></code> by default). This file contains Airflow’s configuration and you
can edit it to change any of the settings. You can also set options with environment variables by using this format:
@@ -258,6 +285,33 @@ its direction.</p>
to monitor your workers. You can use the shortcut command <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">flower</span></code>
to start a Flower web server.</p>
</div>
+<div class="section" id="scaling-out-with-dask">
+<h2>Scaling Out with Dask<a class="headerlink" href="#scaling-out-with-dask" title="Permalink to this headline">�</a></h2>
+<p><code class="docutils literal"><span class="pre">DaskExecutor</span></code> allows you to run Airflow tasks in a Dask Distributed cluster.</p>
+<p>Dask clusters can be run on a single machine or on remote networks. For complete
+details, consult the <a class="reference external" href="https://distributed.readthedocs.io/">Distributed documentation</a>.</p>
+<p>To create a cluster, first start a Scheduler:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># default settings for a local cluster</span>
+<span class="nv">DASK_HOST</span><span class="o">=</span>127.0.0.1
+<span class="nv">DASK_PORT</span><span class="o">=</span>8786
+
+dask-scheduler --host <span class="nv">$DASK_HOST</span> --port <span class="nv">$DASK_PORT</span>
+</pre></div>
+</div>
+<p>Next start at least one Worker on any machine that can connect to the host:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span>dask-worker <span class="nv">$DASK_HOST</span>:<span class="nv">$DASK_PORT</span>
+</pre></div>
+</div>
+<p>Edit your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> to set your executor to <code class="docutils literal"><span class="pre">DaskExecutor</span></code> and provide
+the Dask Scheduler address in the <code class="docutils literal"><span class="pre">[dask]</span></code> section.</p>
+<p>Please note:</p>
+<ul class="simple">
+<li>Each Dask worker must be able to import Airflow and any dependencies you
+require.</li>
+<li>Dask does not support queues. If an Airflow task was created with a queue, a
+warning will be raised but the task will be submitted to the cluster.</li>
+</ul>
+</div>
<div class="section" id="logs">
<h2>Logs<a class="headerlink" href="#logs" title="Permalink to this headline">�</a></h2>
<p>Users can specify a logs folder in <code class="docutils literal"><span class="pre">airflow.cfg</span></code>. By default, it is in
@@ -346,19 +400,37 @@ integrated with upstart</p>
</pre></div>
</div>
</div>
+<div class="section" id="test-mode">
+<h2>Test Mode<a class="headerlink" href="#test-mode" title="Permalink to this headline">�</a></h2>
+<p>Airflow has a fixed set of “test mode” configuration options. You can load these
+at any time by calling <code class="docutils literal"><span class="pre">airflow.configuration.load_test_config()</span></code> (note this
+operation is not reversible!). However, some options (like the DAG_FOLDER) are
+loaded before you have a chance to call load_test_config(). In order to eagerly load
+the test configuration, set test_mode in airflow.cfg:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>tests<span class="o">]</span>
+<span class="nv">unit_test_mode</span> <span class="o">=</span> True
+</pre></div>
+</div>
+<p>Due to Airflow’s automatic environment variable expansion (see <a class="reference internal" href="#setting-options"><span class="std std-ref">Setting Configuration Options</span></a>),
+you can also set the env var <code class="docutils literal"><span class="pre">AIRFLOW__CORE__UNIT_TEST_MODE</span></code> to temporarily overwrite
+airflow.cfg.</p>
+</div>
</div>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
- <a href="ui.html" class="btn btn-neutral float-right" title="UI / Screenshots" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+ <a href="ui.html" class="btn btn-neutral float-right" title="UI / Screenshots" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
- <a href="tutorial.html" class="btn btn-neutral" title="Tutorial" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+ <a href="tutorial.html" class="btn btn-neutral" title="Tutorial" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
</div>
@@ -391,7 +463,8 @@ integrated with upstart</p>
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/faq.html
----------------------------------------------------------------------
diff --git a/faq.html b/faq.html
index 623e37f..7b6c8ab 100644
--- a/faq.html
+++ b/faq.html
@@ -30,9 +30,12 @@
+ <link rel="index" title="Index"
+ href="genindex.html"/>
+ <link rel="search" title="Search" href="search.html"/>
<link rel="top" title="Airflow Documentation" href="index.html"/>
<link rel="next" title="API Reference" href="code.html"/>
- <link rel="prev" title="Security" href="security.html"/>
+ <link rel="prev" title="Integration" href="integration.html"/>
<script src="_static/js/modernizr.min.js"></script>
@@ -41,6 +44,7 @@
<body class="wy-body-for-nav" role="document">
+
<div class="wy-grid-for-nav">
@@ -91,6 +95,8 @@
<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="api.html">Experimental Rest API</a></li>
+<li class="toctree-l1"><a class="reference internal" href="integration.html">Integration</a></li>
<li class="toctree-l1 current"><a class="current reference internal" href="#">FAQ</a><ul>
<li class="toctree-l2"><a class="reference internal" href="#why-isn-t-my-task-getting-scheduled">Why isn’t my task getting scheduled?</a></li>
<li class="toctree-l2"><a class="reference internal" href="#how-do-i-trigger-tasks-based-on-another-task-s-failure">How do I trigger tasks based on another task’s failure?</a></li>
@@ -113,8 +119,10 @@
<nav class="wy-nav-top" role="navigation" aria-label="top navigation">
- <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
- <a href="index.html">Airflow</a>
+
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+
</nav>
@@ -123,23 +131,40 @@
<div class="rst-content">
-
+
+
+
+
+
+
+
+
+
+
+
<div role="navigation" aria-label="breadcrumbs navigation">
+
<ul class="wy-breadcrumbs">
- <li><a href="index.html">Docs</a> »</li>
-
- <li>FAQ</li>
+
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>FAQ</li>
+
+
<li class="wy-breadcrumbs-aside">
-
- <a href="_sources/faq.txt" rel="nofollow"> View page source</a>
+
+ <a href="_sources/faq.rst.txt" rel="nofollow"> View page source</a>
</li>
+
</ul>
+
+
<hr/>
</div>
<div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
@@ -161,6 +186,10 @@ may way to confirm that this works both where the scheduler runs as well
as where the worker runs.</li>
<li>Is your <code class="docutils literal"><span class="pre">start_date</span></code> set properly? The Airflow scheduler triggers the
task soon after the <code class="docutils literal"><span class="pre">start_date</span> <span class="pre">+</span> <span class="pre">scheduler_interval</span></code> is passed.</li>
+<li>Is your <code class="docutils literal"><span class="pre">schedule_interval</span></code> set properly? The default <code class="docutils literal"><span class="pre">schedule_interval</span></code>
+is one day (<code class="docutils literal"><span class="pre">datetime.timedelta(1)</span></code>). You must specify a different <code class="docutils literal"><span class="pre">schedule_interval</span></code>
+directly to the DAG object you instantiate, not as a <code class="docutils literal"><span class="pre">default_param</span></code>, as task instances
+do not override their parent DAG’s <code class="docutils literal"><span class="pre">schedule_interval</span></code>.</li>
<li>Is your <code class="docutils literal"><span class="pre">start_date</span></code> beyond where you can see it in the UI? If you
set your it to some time say 3 months ago, you won’t be able to see
it in the main view in the UI, but you should be able to see it in the
@@ -171,7 +200,7 @@ if you have set <code class="docutils literal"><span class="pre">depends_on_past
needs to have succeeded (except if it is the first run for that task).
Also, if <code class="docutils literal"><span class="pre">wait_for_downstream=True</span></code>, make sure you understand
what it means.
-You can view how these properties are set from the <code class="docutils literal"><span class="pre">Task</span> <span class="pre">Details</span></code>
+You can view how these properties are set from the <code class="docutils literal"><span class="pre">Task</span> <span class="pre">Instance</span> <span class="pre">Details</span></code>
page for your task.</li>
<li>Are the DagRuns you need created and active? A DagRun represents a specific
execution of an entire DAG and has a state (running, success, failed, ...).
@@ -263,7 +292,7 @@ simple dictionary.</p>
command in the queue for it to run remote, on the worker. If using
LocalExecutor, that translates into running it in a subprocess pool.</li>
<li>Local <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span> <span class="pre">--local</span></code>: starts an <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span> <span class="pre">--raw</span></code>
-command (described bellow) as a subprocess and is in charge of
+command (described below) as a subprocess and is in charge of
emitting heartbeats, listening for external kill signals
and ensures some cleanup takes place if the subprocess fails</li>
<li>Raw <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span> <span class="pre">--raw</span></code> runs the actual operator’s execute method and
@@ -274,15 +303,18 @@ performs the actual work</li>
</div>
+ <div class="articleComments">
+
+ </div>
</div>
<footer>
<div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
- <a href="code.html" class="btn btn-neutral float-right" title="API Reference" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+ <a href="code.html" class="btn btn-neutral float-right" title="API Reference" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a>
- <a href="security.html" class="btn btn-neutral" title="Security" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+ <a href="integration.html" class="btn btn-neutral" title="Integration" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a>
</div>
@@ -315,7 +347,8 @@ performs the actual work</li>
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
- HAS_SOURCE: true
+ HAS_SOURCE: true,
+ SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>