You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ma...@apache.org on 2016/06/05 05:23:51 UTC
[01/34] incubator-airflow-site git commit: Initial commit
Repository: incubator-airflow-site
Updated Branches:
refs/heads/asf-site [created] 9e19165ca
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/start.html
----------------------------------------------------------------------
diff --git a/start.html b/start.html
new file mode 100644
index 0000000..6bf0db6
--- /dev/null
+++ b/start.html
@@ -0,0 +1,256 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Quick Start — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="Installation" href="installation.html"/>
+ <link rel="prev" title="License" href="license.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">Quick Start</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#what-s-next">What’s Next?</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>Quick Start</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/start.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="quick-start">
+<h1>Quick Start<a class="headerlink" href="#quick-start" title="Permalink to this headline">�</a></h1>
+<p>The installation is quick and straightforward.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># airflow needs a home, ~/airflow is the default,</span>
+<span class="c1"># but you can lay foundation somewhere else if you prefer</span>
+<span class="c1"># (optional)</span>
+<span class="nb">export</span> <span class="nv">AIRFLOW_HOME</span><span class="o">=</span>~/airflow
+
+<span class="c1"># install from pypi using pip</span>
+pip install airflow
+
+<span class="c1"># initialize the database</span>
+airflow initdb
+
+<span class="c1"># start the web server, default port is 8080</span>
+airflow webserver -p 8080
+</pre></div>
+</div>
+<p>Upon running these commands, Airflow will create the <code class="docutils literal"><span class="pre">$AIRFLOW_HOME</span></code> folder
+and lay an “airflow.cfg” file with defaults that get you going fast. You can
+inspect the file either in <code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code>, or through the UI in
+the <code class="docutils literal"><span class="pre">Admin->Configuration</span></code> menu. The PID file for the webserver will be stored
+in <code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow-webserver.pid</span></code> or in <code class="docutils literal"><span class="pre">/run/airflow/webserver.pid</span></code>
+if started by systemd.</p>
+<p>Out of the box, Airflow uses a sqlite database, which you should outgrow
+fairly quickly since no parallelization is possible using this database
+backend. It works in conjunction with the <code class="docutils literal"><span class="pre">SequentialExecutor</span></code> which will
+only run task instances sequentially. While this is very limiting, it allows
+you to get up and running quickly and take a tour of the UI and the
+command line utilities.</p>
+<p>Here are a few commands that will trigger a few task instances. You should
+be able to see the status of the jobs change in the <code class="docutils literal"><span class="pre">example1</span></code> DAG as you
+run the commands below.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># run your first task instance</span>
+airflow run example_bash_operator runme_0 2015-01-01
+<span class="c1"># run a backfill over 2 days</span>
+airflow backfill example_bash_operator -s 2015-01-01 -e 2015-01-02
+</pre></div>
+</div>
+<div class="section" id="what-s-next">
+<h2>What’s Next?<a class="headerlink" href="#what-s-next" title="Permalink to this headline">�</a></h2>
+<p>From this point, you can head to the <a class="reference internal" href="tutorial.html"><span class="doc">Tutorial</span></a> section for further examples or the <a class="reference internal" href="configuration.html"><span class="doc">Configuration</span></a> section if you’re ready to get your hands dirty.</p>
+</div>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="installation.html" class="btn btn-neutral float-right" title="Installation" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="license.html" class="btn btn-neutral" title="License" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/tutorial.html
----------------------------------------------------------------------
diff --git a/tutorial.html b/tutorial.html
new file mode 100644
index 0000000..d0adb06
--- /dev/null
+++ b/tutorial.html
@@ -0,0 +1,622 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Tutorial — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="Configuration" href="configuration.html"/>
+ <link rel="prev" title="Installation" href="installation.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">Tutorial</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#example-pipeline-definition">Example Pipeline definition</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#it-s-a-dag-definition-file">It’s a DAG definition file</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#importing-modules">Importing Modules</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#default-arguments">Default Arguments</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#instantiate-a-dag">Instantiate a DAG</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#tasks">Tasks</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#templating-with-jinja">Templating with Jinja</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#setting-up-dependencies">Setting up Dependencies</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#recap">Recap</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#testing">Testing</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#running-the-script">Running the Script</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#command-line-metadata-validation">Command Line Metadata Validation</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#id1">Testing</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#backfill">Backfill</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="#what-s-next">What’s Next?</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>Tutorial</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/tutorial.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="tutorial">
+<h1>Tutorial<a class="headerlink" href="#tutorial" title="Permalink to this headline">�</a></h1>
+<p>This tutorial walks you through some of the fundamental Airflow concepts,
+objects, and their usage while writing your first pipeline.</p>
+<div class="section" id="example-pipeline-definition">
+<h2>Example Pipeline definition<a class="headerlink" href="#example-pipeline-definition" title="Permalink to this headline">�</a></h2>
+<p>Here is an example of a basic pipeline definition. Do not worry if this looks
+complicated, a line by line explanation follows below.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="sd">"""</span>
+<span class="sd">Code that goes along with the Airflow tutorial located at:</span>
+<span class="sd">https://github.com/airbnb/airflow/blob/master/airflow/example_dags/tutorial.py</span>
+<span class="sd">"""</span>
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">DAG</span>
+<span class="kn">from</span> <span class="nn">airflow.operators</span> <span class="k">import</span> <span class="n">BashOperator</span>
+<span class="kn">from</span> <span class="nn">datetime</span> <span class="k">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timedelta</span>
+
+
+<span class="n">default_args</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'owner'</span><span class="p">:</span> <span class="s1">'airflow'</span><span class="p">,</span>
+ <span class="s1">'depends_on_past'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
+ <span class="s1">'start_date'</span><span class="p">:</span> <span class="n">datetime</span><span class="p">(</span><span class="mi">2015</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>
+ <span class="s1">'email'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'airflow@airflow.com'</span><span class="p">],</span>
+ <span class="s1">'email_on_failure'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
+ <span class="s1">'email_on_retry'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
+ <span class="s1">'retries'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
+ <span class="s1">'retry_delay'</span><span class="p">:</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">minutes</span><span class="o">=</span><span class="mi">5</span><span class="p">),</span>
+ <span class="c1"># 'queue': 'bash_queue',</span>
+ <span class="c1"># 'pool': 'backfill',</span>
+ <span class="c1"># 'priority_weight': 10,</span>
+ <span class="c1"># 'end_date': datetime(2016, 1, 1),</span>
+<span class="p">}</span>
+
+<span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'tutorial'</span><span class="p">,</span> <span class="n">default_args</span><span class="o">=</span><span class="n">default_args</span><span class="p">)</span>
+
+<span class="c1"># t1, t2 and t3 are examples of tasks created by instatiating operators</span>
+<span class="n">t1</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="s1">'print_date'</span><span class="p">,</span>
+ <span class="n">bash_command</span><span class="o">=</span><span class="s1">'date'</span><span class="p">,</span>
+ <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+
+<span class="n">t2</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="s1">'sleep'</span><span class="p">,</span>
+ <span class="n">bash_command</span><span class="o">=</span><span class="s1">'sleep 5'</span><span class="p">,</span>
+ <span class="n">retries</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span>
+ <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+
+<span class="n">templated_command</span> <span class="o">=</span> <span class="s2">"""</span>
+<span class="s2"> {</span><span class="si">% f</span><span class="s2">or i in range(5) %}</span>
+<span class="s2"> echo "{{ ds }}"</span>
+<span class="s2"> echo "{{ macros.ds_add(ds, 7)}}"</span>
+<span class="s2"> echo "{{ params.my_param }}"</span>
+<span class="s2"> {</span><span class="si">% e</span><span class="s2">ndfor %}</span>
+<span class="s2">"""</span>
+
+<span class="n">t3</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="s1">'templated'</span><span class="p">,</span>
+ <span class="n">bash_command</span><span class="o">=</span><span class="n">templated_command</span><span class="p">,</span>
+ <span class="n">params</span><span class="o">=</span><span class="p">{</span><span class="s1">'my_param'</span><span class="p">:</span> <span class="s1">'Parameter I passed in'</span><span class="p">},</span>
+ <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+
+<span class="n">t2</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span>
+<span class="n">t3</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="it-s-a-dag-definition-file">
+<h2>It’s a DAG definition file<a class="headerlink" href="#it-s-a-dag-definition-file" title="Permalink to this headline">�</a></h2>
+<p>One thing to wrap your head around (it may not be very intuitive for everyone
+at first) is that this Airflow Python script is really
+just a configuration file specifying the DAG’s structure as code.
+The actual tasks defined here will run in a different context from
+the context of this script. Different tasks run on different workers
+at different points in time, which means that this script cannot be used
+to cross communicate between tasks. Note that for this
+purpose we have a more advanced feature called <code class="docutils literal"><span class="pre">XCom</span></code>.</p>
+<p>People sometimes think of the DAG definition file as a place where they
+can do some actual data processing - that is not the case at all!
+The script’s purpose is to define a DAG object. It needs to evaluate
+quickly (seconds, not minutes) since the scheduler will execute it
+periodically to reflect the changes if any.</p>
+</div>
+<div class="section" id="importing-modules">
+<h2>Importing Modules<a class="headerlink" href="#importing-modules" title="Permalink to this headline">�</a></h2>
+<p>An Airflow pipeline is just a Python script that happens to define an
+Airflow DAG object. Let’s start by importing the libraries we will need.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="c1"># The DAG object; we'll need this to instantiate a DAG</span>
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">DAG</span>
+
+<span class="c1"># Operators; we need this to operate!</span>
+<span class="kn">from</span> <span class="nn">airflow.operators</span> <span class="k">import</span> <span class="n">BashOperator</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="default-arguments">
+<h2>Default Arguments<a class="headerlink" href="#default-arguments" title="Permalink to this headline">�</a></h2>
+<p>We’re about to create a DAG and some tasks, and we have the choice to
+explicitly pass a set of arguments to each task’s constructor
+(which would become redundant), or (better!) we can define a dictionary
+of default parameters that we can use when creating tasks.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">datetime</span> <span class="k">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timedelta</span>
+
+<span class="n">default_args</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'owner'</span><span class="p">:</span> <span class="s1">'airflow'</span><span class="p">,</span>
+ <span class="s1">'depends_on_past'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
+ <span class="s1">'start_date'</span><span class="p">:</span> <span class="n">datetime</span><span class="p">(</span><span class="mi">2015</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>
+ <span class="s1">'email'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'airflow@airflow.com'</span><span class="p">],</span>
+ <span class="s1">'email_on_failure'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
+ <span class="s1">'email_on_retry'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
+ <span class="s1">'retries'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
+ <span class="s1">'retry_delay'</span><span class="p">:</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">minutes</span><span class="o">=</span><span class="mi">5</span><span class="p">),</span>
+ <span class="c1"># 'queue': 'bash_queue',</span>
+ <span class="c1"># 'pool': 'backfill',</span>
+ <span class="c1"># 'priority_weight': 10,</span>
+ <span class="c1"># 'end_date': datetime(2016, 1, 1),</span>
+<span class="p">}</span>
+</pre></div>
+</div>
+<p>For more information about the BaseOperator’s parameters and what they do,
+refer to the :py:class:<code class="docutils literal"><span class="pre">airflow.models.BaseOperator</span></code> documentation.</p>
+<p>Also, note that you could easily define different sets of arguments that
+would serve different purposes. An example of that would be to have
+different settings between a production and development environment.</p>
+</div>
+<div class="section" id="instantiate-a-dag">
+<h2>Instantiate a DAG<a class="headerlink" href="#instantiate-a-dag" title="Permalink to this headline">�</a></h2>
+<p>We’ll need a DAG object to nest our tasks into. Here we pass a string
+that defines the <code class="docutils literal"><span class="pre">dag_id</span></code>, which serves as a unique identifier for your DAG.
+We also pass the default argument dictionary that we just defined and
+define a <code class="docutils literal"><span class="pre">schedule_interval</span></code> of 1 day for the DAG.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span>
+ <span class="s1">'tutorial'</span><span class="p">,</span> <span class="n">default_args</span><span class="o">=</span><span class="n">default_args</span><span class="p">,</span> <span class="n">schedule_interval</span><span class="o">=</span><span class="n">timedelta</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="tasks">
+<h2>Tasks<a class="headerlink" href="#tasks" title="Permalink to this headline">�</a></h2>
+<p>Tasks are generated when instantiating operator objects. An object
+instantiated from an operator is called a constructor. The first argument
+<code class="docutils literal"><span class="pre">task_id</span></code> acts as a unique identifier for the task.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">t1</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="s1">'print_date'</span><span class="p">,</span>
+ <span class="n">bash_command</span><span class="o">=</span><span class="s1">'date'</span><span class="p">,</span>
+ <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+
+<span class="n">t2</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="s1">'sleep'</span><span class="p">,</span>
+ <span class="n">bash_command</span><span class="o">=</span><span class="s1">'sleep 5'</span><span class="p">,</span>
+ <span class="n">retries</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span>
+ <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>Notice how we pass a mix of operator specific arguments (<code class="docutils literal"><span class="pre">bash_command</span></code>) and
+an argument common to all operators (<code class="docutils literal"><span class="pre">retries</span></code>) inherited
+from BaseOperator to the operator’s constructor. This is simpler than
+passing every argument for every constructor call. Also, notice that in
+the second task we override the <code class="docutils literal"><span class="pre">retries</span></code> parameter with <code class="docutils literal"><span class="pre">3</span></code>.</p>
+<p>The precedence rules for a task are as follows:</p>
+<ol class="arabic simple">
+<li>Explicitly passed arguments</li>
+<li>Values that exist in the <code class="docutils literal"><span class="pre">default_args</span></code> dictionary</li>
+<li>The operator’s default value, if one exists</li>
+</ol>
+<p>A task must include or inherit the arguments <code class="docutils literal"><span class="pre">task_id</span></code> and <code class="docutils literal"><span class="pre">owner</span></code>,
+otherwise Airflow will raise an exception.</p>
+</div>
+<div class="section" id="templating-with-jinja">
+<h2>Templating with Jinja<a class="headerlink" href="#templating-with-jinja" title="Permalink to this headline">�</a></h2>
+<p>Airflow leverages the power of
+<a class="reference external" href="http://jinja.pocoo.org/docs/dev/">Jinja Templating</a> and provides
+the pipeline author
+with a set of built-in parameters and macros. Airflow also provides
+hooks for the pipeline author to define their own parameters, macros and
+templates.</p>
+<p>This tutorial barely scratches the surface of what you can do with
+templating in Airflow, but the goal of this section is to let you know
+this feature exists, get you familiar with double curly brackets, and
+point to the most common template variable: <code class="docutils literal"><span class="pre">{{</span> <span class="pre">ds</span> <span class="pre">}}</span></code>.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">templated_command</span> <span class="o">=</span> <span class="s2">"""</span>
+<span class="s2"> {</span><span class="si">% f</span><span class="s2">or i in range(5) %}</span>
+<span class="s2"> echo "{{ ds }}"</span>
+<span class="s2"> echo "{{ macros.ds_add(ds, 7) }}"</span>
+<span class="s2"> echo "{{ params.my_param }}"</span>
+<span class="s2"> {</span><span class="si">% e</span><span class="s2">ndfor %}</span>
+<span class="s2">"""</span>
+
+<span class="n">t3</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="s1">'templated'</span><span class="p">,</span>
+ <span class="n">bash_command</span><span class="o">=</span><span class="n">templated_command</span><span class="p">,</span>
+ <span class="n">params</span><span class="o">=</span><span class="p">{</span><span class="s1">'my_param'</span><span class="p">:</span> <span class="s1">'Parameter I passed in'</span><span class="p">},</span>
+ <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>Notice that the <code class="docutils literal"><span class="pre">templated_command</span></code> contains code logic in <code class="docutils literal"><span class="pre">{%</span> <span class="pre">%}</span></code> blocks,
+references parameters like <code class="docutils literal"><span class="pre">{{</span> <span class="pre">ds</span> <span class="pre">}}</span></code>, calls a function as in
+<code class="docutils literal"><span class="pre">{{</span> <span class="pre">macros.ds_add(ds,</span> <span class="pre">7)}}</span></code>, and references a user-defined parameter
+in <code class="docutils literal"><span class="pre">{{</span> <span class="pre">params.my_param</span> <span class="pre">}}</span></code>.</p>
+<p>The <code class="docutils literal"><span class="pre">params</span></code> hook in <code class="docutils literal"><span class="pre">BaseOperator</span></code> allows you to pass a dictionary of
+parameters and/or objects to your templates. Please take the time
+to understand how the parameter <code class="docutils literal"><span class="pre">my_param</span></code> makes it through to the template.</p>
+<p>Files can also be passed to the <code class="docutils literal"><span class="pre">bash_command</span></code> argument, like
+<code class="docutils literal"><span class="pre">bash_command='templated_command.sh'</span></code>, where the file location is relative to
+the directory containing the pipeline file (<code class="docutils literal"><span class="pre">tutorial.py</span></code> in this case). This
+may be desirable for many reasons, like separating your script’s logic and
+pipeline code, allowing for proper code highlighting in files composed in
+different languages, and general flexibility in structuring pipelines. It is
+also possible to define your <code class="docutils literal"><span class="pre">template_searchpath</span></code> as pointing to any folder
+locations in the DAG constructor call.</p>
+<p>For more information on the variables and macros that can be referenced
+in templates, make sure to read through the <a class="reference internal" href="code.html#macros"><span class="std std-ref">Macros</span></a> section</p>
+</div>
+<div class="section" id="setting-up-dependencies">
+<h2>Setting up Dependencies<a class="headerlink" href="#setting-up-dependencies" title="Permalink to this headline">�</a></h2>
+<p>We have two simple tasks that do not depend on each other. Here’s a few ways
+you can define dependencies between them:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">t2</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span>
+
+<span class="c1"># This means that t2 will depend on t1</span>
+<span class="c1"># running successfully to run</span>
+<span class="c1"># It is equivalent to</span>
+<span class="c1"># t1.set_downstream(t2)</span>
+
+<span class="n">t3</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span>
+
+<span class="c1"># all of this is equivalent to</span>
+<span class="c1"># dag.set_dependency('print_date', 'sleep')</span>
+<span class="c1"># dag.set_dependency('print_date', 'templated')</span>
+</pre></div>
+</div>
+<p>Note that when executing your script, Airflow will raise exceptions when
+it finds cycles in your DAG or when a dependency is referenced more
+than once.</p>
+</div>
+<div class="section" id="recap">
+<h2>Recap<a class="headerlink" href="#recap" title="Permalink to this headline">�</a></h2>
+<p>Alright, so we have a pretty basic DAG. At this point your code should look
+something like this:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="sd">"""</span>
+<span class="sd">Code that goes along with the Airflow located at:</span>
+<span class="sd">http://airflow.readthedocs.org/en/latest/tutorial.html</span>
+<span class="sd">"""</span>
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="k">import</span> <span class="n">DAG</span>
+<span class="kn">from</span> <span class="nn">airflow.operators</span> <span class="k">import</span> <span class="n">BashOperator</span>
+<span class="kn">from</span> <span class="nn">datetime</span> <span class="k">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timedelta</span>
+
+
+<span class="n">default_args</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'owner'</span><span class="p">:</span> <span class="s1">'airflow'</span><span class="p">,</span>
+ <span class="s1">'depends_on_past'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
+ <span class="s1">'start_date'</span><span class="p">:</span> <span class="n">datetime</span><span class="p">(</span><span class="mi">2015</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>
+ <span class="s1">'email'</span><span class="p">:</span> <span class="p">[</span><span class="s1">'airflow@airflow.com'</span><span class="p">],</span>
+ <span class="s1">'email_on_failure'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
+ <span class="s1">'email_on_retry'</span><span class="p">:</span> <span class="kc">False</span><span class="p">,</span>
+ <span class="s1">'retries'</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
+ <span class="s1">'retry_delay'</span><span class="p">:</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">minutes</span><span class="o">=</span><span class="mi">5</span><span class="p">),</span>
+ <span class="c1"># 'queue': 'bash_queue',</span>
+ <span class="c1"># 'pool': 'backfill',</span>
+ <span class="c1"># 'priority_weight': 10,</span>
+ <span class="c1"># 'end_date': datetime(2016, 1, 1),</span>
+<span class="p">}</span>
+
+<span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span>
+ <span class="s1">'tutorial'</span><span class="p">,</span> <span class="n">default_args</span><span class="o">=</span><span class="n">default_args</span><span class="p">,</span> <span class="n">schedule_interval</span><span class="o">=</span><span class="n">timedelta</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
+
+<span class="c1"># t1, t2 and t3 are examples of tasks created by instatiating operators</span>
+<span class="n">t1</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="s1">'print_date'</span><span class="p">,</span>
+ <span class="n">bash_command</span><span class="o">=</span><span class="s1">'date'</span><span class="p">,</span>
+ <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+
+<span class="n">t2</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="s1">'sleep'</span><span class="p">,</span>
+ <span class="n">bash_command</span><span class="o">=</span><span class="s1">'sleep 5'</span><span class="p">,</span>
+ <span class="n">retries</span><span class="o">=</span><span class="mi">3</span><span class="p">,</span>
+ <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+
+<span class="n">templated_command</span> <span class="o">=</span> <span class="s2">"""</span>
+<span class="s2"> {</span><span class="si">% f</span><span class="s2">or i in range(5) %}</span>
+<span class="s2"> echo "{{ ds }}"</span>
+<span class="s2"> echo "{{ macros.ds_add(ds, 7)}}"</span>
+<span class="s2"> echo "{{ params.my_param }}"</span>
+<span class="s2"> {</span><span class="si">% e</span><span class="s2">ndfor %}</span>
+<span class="s2">"""</span>
+
+<span class="n">t3</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="s1">'templated'</span><span class="p">,</span>
+ <span class="n">bash_command</span><span class="o">=</span><span class="n">templated_command</span><span class="p">,</span>
+ <span class="n">params</span><span class="o">=</span><span class="p">{</span><span class="s1">'my_param'</span><span class="p">:</span> <span class="s1">'Parameter I passed in'</span><span class="p">},</span>
+ <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+
+<span class="n">t2</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span>
+<span class="n">t3</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">t1</span><span class="p">)</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="testing">
+<h2>Testing<a class="headerlink" href="#testing" title="Permalink to this headline">�</a></h2>
+<div class="section" id="running-the-script">
+<h3>Running the Script<a class="headerlink" href="#running-the-script" title="Permalink to this headline">�</a></h3>
+<p>Time to run some tests. First let’s make sure that the pipeline
+parses. Let’s assume we’re saving the code from the previous step in
+<code class="docutils literal"><span class="pre">tutorial.py</span></code> in the DAGs folder referenced in your <code class="docutils literal"><span class="pre">airflow.cfg</span></code>.
+The default location for your DAGs is <code class="docutils literal"><span class="pre">~/airflow/dags</span></code>.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span>python ~/airflow/dags/tutorial.py
+</pre></div>
+</div>
+<p>If the script does not raise an exception it means that you haven’t done
+anything horribly wrong, and that your Airflow environment is somewhat
+sound.</p>
+</div>
+<div class="section" id="command-line-metadata-validation">
+<h3>Command Line Metadata Validation<a class="headerlink" href="#command-line-metadata-validation" title="Permalink to this headline">�</a></h3>
+<p>Let’s run a few commands to validate this script further.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># print the list of active DAGs</span>
+airflow list_dags
+
+<span class="c1"># prints the list of tasks the "tutorial" dag_id</span>
+airflow list_tasks tutorial
+
+<span class="c1"># prints the hierarchy of tasks in the tutorial DAG</span>
+airflow list_tasks tutorial --tree
+</pre></div>
+</div>
+</div>
+<div class="section" id="id1">
+<h3>Testing<a class="headerlink" href="#id1" title="Permalink to this headline">�</a></h3>
+<p>Let’s test by running the actual task instances on a specific date. The
+date specified in this context is an <code class="docutils literal"><span class="pre">execution_date</span></code>, which simulates the
+scheduler running your task or dag at a specific date + time:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># command layout: command subcommand dag_id task_id date</span>
+
+<span class="c1"># testing print_date</span>
+airflow <span class="nb">test</span> tutorial print_date 2015-06-01
+
+<span class="c1"># testing sleep</span>
+airflow <span class="nb">test</span> tutorial sleep 2015-06-01
+</pre></div>
+</div>
+<p>Now remember what we did with templating earlier? See how this template
+gets rendered and executed by running this command:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># testing templated</span>
+airflow <span class="nb">test</span> tutorial templated 2015-06-01
+</pre></div>
+</div>
+<p>This should result in displaying a verbose log of events and ultimately
+running your bash command and printing the result.</p>
+<p>Note that the <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">test</span></code> command runs task instances locally, outputs
+their log to stdout (on screen), doesn’t bother with dependencies, and
+doesn’t communicate state (running, success, failed, ...) to the database.
+It simply allows testing a single task instance.</p>
+</div>
+<div class="section" id="backfill">
+<h3>Backfill<a class="headerlink" href="#backfill" title="Permalink to this headline">�</a></h3>
+<p>Everything looks like it’s running fine so let’s run a backfill.
+<code class="docutils literal"><span class="pre">backfill</span></code> will respect your dependencies, emit logs into files and talk to
+the database to record status. If you do have a webserver up, you’ll be able
+to track the progress. <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">webserver</span></code> will start a web server if you
+are interested in tracking the progress visually as your backfill progresses.</p>
+<p>Note that if you use <code class="docutils literal"><span class="pre">depends_on_past=True</span></code>, individual task instances
+will depend on the success of the preceding task instance, except for the
+start_date specified itself, for which this dependency is disregarded.</p>
+<p>The date range in this context is a <code class="docutils literal"><span class="pre">start_date</span></code> and optionally an <code class="docutils literal"><span class="pre">end_date</span></code>,
+which are used to populate the run schedule with task instances from this dag.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># optional, start a web server in debug mode in the background</span>
+<span class="c1"># airflow webserver --debug &</span>
+
+<span class="c1"># start your backfill on a date range</span>
+airflow backfill tutorial -s 2015-06-01 -e 2015-06-07
+</pre></div>
+</div>
+</div>
+</div>
+<div class="section" id="what-s-next">
+<h2>What’s Next?<a class="headerlink" href="#what-s-next" title="Permalink to this headline">�</a></h2>
+<p>That’s it, you’ve written, tested and backfilled your very first Airflow
+pipeline. Merging your code into a code repository that has a master scheduler
+running against it should get it to get triggered and run every day.</p>
+<p>Here’s a few things you might want to do next:</p>
+<ul>
+<li><p class="first">Take an in-depth tour of the UI - click all the things!</p>
+</li>
+<li><p class="first">Keep reading the docs! Especially the sections on:</p>
+<blockquote>
+<div><ul class="simple">
+<li>Command line interface</li>
+<li>Operators</li>
+<li>Macros</li>
+</ul>
+</div></blockquote>
+</li>
+<li><p class="first">Write your first pipeline!</p>
+</li>
+</ul>
+</div>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="configuration.html" class="btn btn-neutral float-right" title="Configuration" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="installation.html" class="btn btn-neutral" title="Installation" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/ui.html
----------------------------------------------------------------------
diff --git a/ui.html b/ui.html
new file mode 100644
index 0000000..df7f6f7
--- /dev/null
+++ b/ui.html
@@ -0,0 +1,296 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>UI / Screenshots — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="Concepts" href="concepts.html"/>
+ <link rel="prev" title="Configuration" href="configuration.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">UI / Screenshots</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#dags-view">DAGs View</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#tree-view">Tree View</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#graph-view">Graph View</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#variable-view">Variable View</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#gantt-chart">Gantt Chart</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#task-duration">Task Duration</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#code-view">Code View</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#task-instance-context-menu">Task Instance Context Menu</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>UI / Screenshots</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/ui.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="ui-screenshots">
+<h1>UI / Screenshots<a class="headerlink" href="#ui-screenshots" title="Permalink to this headline">�</a></h1>
+<p>The Airflow UI make it easy to monitor and troubleshoot your data pipelines.
+Here’s a quick overview of some of the features and visualizations you
+can find in the Airflow UI.</p>
+<div class="section" id="dags-view">
+<h2>DAGs View<a class="headerlink" href="#dags-view" title="Permalink to this headline">�</a></h2>
+<p>List of the DAGs in your environment, and a set of shortcuts to useful pages.
+You can see exactly how many tasks succeeded, failed, or are currently
+running at a glance.</p>
+<hr class="docutils" />
+<img alt="_images/dags.png" src="_images/dags.png" />
+</div>
+<hr class="docutils" />
+<div class="section" id="tree-view">
+<h2>Tree View<a class="headerlink" href="#tree-view" title="Permalink to this headline">�</a></h2>
+<p>A tree representation of the DAG that spans across time. If a pipeline is
+late, you can quickly see where the different steps are and identify
+the blocking ones.</p>
+<hr class="docutils" />
+<img alt="_images/tree.png" src="_images/tree.png" />
+</div>
+<hr class="docutils" />
+<div class="section" id="graph-view">
+<h2>Graph View<a class="headerlink" href="#graph-view" title="Permalink to this headline">�</a></h2>
+<p>The graph view is perhaps the most comprehensive. Visualize your DAG’s
+dependencies and their current status for a specific run.</p>
+<hr class="docutils" />
+<img alt="_images/graph.png" src="_images/graph.png" />
+</div>
+<hr class="docutils" />
+<div class="section" id="variable-view">
+<h2>Variable View<a class="headerlink" href="#variable-view" title="Permalink to this headline">�</a></h2>
+<p>The variable view allows you to list, create, edit or delete the key-value pair
+of a variable used during jobs. Value of a variable will be hidden if the key contains
+any words in (‘password’, ‘secret’, ‘passwd’, ‘authorization’, ‘api_key’, ‘apikey’, ‘access_token’)
+by default, but can be configured to show in clear-text.</p>
+<hr class="docutils" />
+<img alt="_images/variable_hidden.png" src="_images/variable_hidden.png" />
+</div>
+<hr class="docutils" />
+<div class="section" id="gantt-chart">
+<h2>Gantt Chart<a class="headerlink" href="#gantt-chart" title="Permalink to this headline">�</a></h2>
+<p>The Gantt chart lets you analyse task duration and overlap. You can quickly
+identify bottlenecks and where the bulk of the time is spent for specific
+DAG runs.</p>
+<hr class="docutils" />
+<img alt="_images/gantt.png" src="_images/gantt.png" />
+</div>
+<hr class="docutils" />
+<div class="section" id="task-duration">
+<h2>Task Duration<a class="headerlink" href="#task-duration" title="Permalink to this headline">�</a></h2>
+<p>The duration of your different tasks over the past N runs. This view lets
+you find outliers and quickly understand where the time is spent in your
+DAG over many runs.</p>
+<hr class="docutils" />
+<img alt="_images/duration.png" src="_images/duration.png" />
+</div>
+<hr class="docutils" />
+<div class="section" id="code-view">
+<h2>Code View<a class="headerlink" href="#code-view" title="Permalink to this headline">�</a></h2>
+<p>Transparency is everything. While the code for your pipeline is in source
+control, this is a quick way to get to the code that generates the DAG and
+provide yet more context.</p>
+<hr class="docutils" />
+<img alt="_images/code.png" src="_images/code.png" />
+</div>
+<hr class="docutils" />
+<div class="section" id="task-instance-context-menu">
+<h2>Task Instance Context Menu<a class="headerlink" href="#task-instance-context-menu" title="Permalink to this headline">�</a></h2>
+<p>From the pages seen above (tree view, graph view, gantt, ...), it is always
+possible to click on a task instance, and get to this rich context menu
+that can take you to more detailed metadata, and perform some actions.</p>
+<hr class="docutils" />
+<img alt="_images/context.png" src="_images/context.png" />
+</div>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="concepts.html" class="btn btn-neutral float-right" title="Concepts" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="configuration.html" class="btn btn-neutral" title="Configuration" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[32/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/airflow/macros/hive.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/macros/hive.html b/_modules/airflow/macros/hive.html
new file mode 100644
index 0000000..cfcc1a0
--- /dev/null
+++ b/_modules/airflow/macros/hive.html
@@ -0,0 +1,298 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.macros.hive — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../../../index.html"/>
+ <link rel="up" title="airflow.macros" href="../macros.html"/>
+
+
+ <script src="../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../index.html">Module code</a> »</li>
+
+ <li><a href="../macros.html">airflow.macros</a> »</li>
+
+ <li>airflow.macros.hive</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.macros.hive</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">datetime</span>
+
+
+<div class="viewcode-block" id="max_partition"><a class="viewcode-back" href="../../../code.html#airflow.macros.hive.max_partition">[docs]</a><span class="k">def</span> <span class="nf">max_partition</span><span class="p">(</span>
+ <span class="n">table</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="s2">"default"</span><span class="p">,</span> <span class="n">field</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="nb">filter</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">metastore_conn_id</span><span class="o">=</span><span class="s1">'metastore_default'</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> Gets the max partition for a table.</span>
+
+<span class="sd"> :param schema: The hive schema the table lives in</span>
+<span class="sd"> :type schema: string</span>
+<span class="sd"> :param table: The hive table you are interested in, supports the dot</span>
+<span class="sd"> notation as in "my_database.my_table", if a dot is found,</span>
+<span class="sd"> the schema param is disregarded</span>
+<span class="sd"> :type table: string</span>
+<span class="sd"> :param hive_conn_id: The hive connection you are interested in.</span>
+<span class="sd"> If your default is set you don't need to use this parameter.</span>
+<span class="sd"> :type hive_conn_id: string</span>
+<span class="sd"> :param filter: filter on a subset of partition as in</span>
+<span class="sd"> `sub_part='specific_value'`</span>
+<span class="sd"> :type filter: string</span>
+<span class="sd"> :param field: the field to get the max value from. If there's only</span>
+<span class="sd"> one partition field, this will be inferred</span>
+
+<span class="sd"> >>> max_partition('airflow.static_babynames_partitioned')</span>
+<span class="sd"> '2015-01-01'</span>
+<span class="sd"> '''</span>
+ <span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">HiveMetastoreHook</span>
+ <span class="k">if</span> <span class="s1">'.'</span> <span class="ow">in</span> <span class="n">table</span><span class="p">:</span>
+ <span class="n">schema</span><span class="p">,</span> <span class="n">table</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span>
+ <span class="n">hh</span> <span class="o">=</span> <span class="n">HiveMetastoreHook</span><span class="p">(</span><span class="n">metastore_conn_id</span><span class="o">=</span><span class="n">metastore_conn_id</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">hh</span><span class="o">.</span><span class="n">max_partition</span><span class="p">(</span>
+ <span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">,</span> <span class="n">table_name</span><span class="o">=</span><span class="n">table</span><span class="p">,</span> <span class="n">field</span><span class="o">=</span><span class="n">field</span><span class="p">,</span> <span class="nb">filter</span><span class="o">=</span><span class="nb">filter</span><span class="p">)</span></div>
+
+
+<span class="k">def</span> <span class="nf">_closest_date</span><span class="p">(</span><span class="n">target_dt</span><span class="p">,</span> <span class="n">date_list</span><span class="p">,</span> <span class="n">before_target</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> This function finds the date in a list closest to the target date.</span>
+<span class="sd"> An optional parameter can be given to get the closest before or after.</span>
+
+<span class="sd"> :param target_dt: The target date</span>
+<span class="sd"> :type target_dt: datetime.date</span>
+<span class="sd"> :param date_list: The list of dates to search</span>
+<span class="sd"> :type date_list: datetime.date list</span>
+<span class="sd"> :param before_target: closest before or after the target</span>
+<span class="sd"> :type before_target: bool or None</span>
+<span class="sd"> :returns: The closest date</span>
+<span class="sd"> :rtype: datetime.date or None</span>
+<span class="sd"> '''</span>
+ <span class="n">fb</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">d</span><span class="p">:</span> <span class="n">d</span> <span class="o">-</span> <span class="n">target_dt</span> <span class="k">if</span> <span class="n">d</span> <span class="o">>=</span> <span class="n">target_dt</span> <span class="k">else</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="o">.</span><span class="n">max</span>
+ <span class="n">fa</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">d</span><span class="p">:</span> <span class="n">d</span> <span class="o">-</span> <span class="n">target_dt</span> <span class="k">if</span> <span class="n">d</span> <span class="o"><=</span> <span class="n">target_dt</span> <span class="k">else</span> <span class="n">datetime</span><span class="o">.</span><span class="n">timedelta</span><span class="o">.</span><span class="n">min</span>
+ <span class="n">fnone</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">d</span><span class="p">:</span> <span class="n">target_dt</span> <span class="o">-</span> <span class="n">d</span> <span class="k">if</span> <span class="n">d</span> <span class="o"><</span> <span class="n">target_dt</span> <span class="k">else</span> <span class="n">d</span> <span class="o">-</span> <span class="n">target_dt</span>
+ <span class="k">if</span> <span class="n">before_target</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
+ <span class="k">return</span> <span class="nb">min</span><span class="p">(</span><span class="n">date_list</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="n">fnone</span><span class="p">)</span><span class="o">.</span><span class="n">date</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">before_target</span><span class="p">:</span>
+ <span class="k">return</span> <span class="nb">min</span><span class="p">(</span><span class="n">date_list</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="n">fb</span><span class="p">)</span><span class="o">.</span><span class="n">date</span><span class="p">()</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span> <span class="nb">min</span><span class="p">(</span><span class="n">date_list</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="n">fa</span><span class="p">)</span><span class="o">.</span><span class="n">date</span><span class="p">()</span>
+
+
+<div class="viewcode-block" id="closest_ds_partition"><a class="viewcode-back" href="../../../code.html#airflow.macros.hive.closest_ds_partition">[docs]</a><span class="k">def</span> <span class="nf">closest_ds_partition</span><span class="p">(</span>
+ <span class="n">table</span><span class="p">,</span> <span class="n">ds</span><span class="p">,</span> <span class="n">before</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="s2">"default"</span><span class="p">,</span>
+ <span class="n">metastore_conn_id</span><span class="o">=</span><span class="s1">'metastore_default'</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> This function finds the date in a list closest to the target date.</span>
+<span class="sd"> An optional parameter can be given to get the closest before or after.</span>
+
+<span class="sd"> :param table: A hive table name</span>
+<span class="sd"> :type table: str</span>
+<span class="sd"> :param ds: A datestamp ``%Y-%m-%d`` e.g. ``yyyy-mm-dd``</span>
+<span class="sd"> :type ds: datetime.date list</span>
+<span class="sd"> :param before: closest before (True), after (False) or either side of ds</span>
+<span class="sd"> :type before: bool or None</span>
+<span class="sd"> :returns: The closest date</span>
+<span class="sd"> :rtype: str or None</span>
+
+<span class="sd"> >>> tbl = 'airflow.static_babynames_partitioned'</span>
+<span class="sd"> >>> closest_ds_partition(tbl, '2015-01-02')</span>
+<span class="sd"> '2015-01-01'</span>
+<span class="sd"> '''</span>
+ <span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">HiveMetastoreHook</span>
+ <span class="k">if</span> <span class="s1">'.'</span> <span class="ow">in</span> <span class="n">table</span><span class="p">:</span>
+ <span class="n">schema</span><span class="p">,</span> <span class="n">table</span> <span class="o">=</span> <span class="n">table</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span>
+ <span class="n">hh</span> <span class="o">=</span> <span class="n">HiveMetastoreHook</span><span class="p">(</span><span class="n">metastore_conn_id</span><span class="o">=</span><span class="n">metastore_conn_id</span><span class="p">)</span>
+ <span class="n">partitions</span> <span class="o">=</span> <span class="n">hh</span><span class="o">.</span><span class="n">get_partitions</span><span class="p">(</span><span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">,</span> <span class="n">table_name</span><span class="o">=</span><span class="n">table</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">partitions</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">None</span>
+ <span class="n">part_vals</span> <span class="o">=</span> <span class="p">[</span><span class="nb">list</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">values</span><span class="p">())[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">partitions</span><span class="p">]</span>
+ <span class="k">if</span> <span class="n">ds</span> <span class="ow">in</span> <span class="n">part_vals</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">ds</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">parts</span> <span class="o">=</span> <span class="p">[</span><span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">pv</span><span class="p">,</span> <span class="s1">'%Y-%m-</span><span class="si">%d</span><span class="s1">'</span><span class="p">)</span>
+ <span class="k">for</span> <span class="n">pv</span> <span class="ow">in</span> <span class="n">part_vals</span><span class="p">]</span>
+ <span class="n">target_dt</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">ds</span><span class="p">,</span> <span class="s1">'%Y-%m-</span><span class="si">%d</span><span class="s1">'</span><span class="p">)</span>
+ <span class="n">closest_ds</span> <span class="o">=</span> <span class="n">_closest_date</span><span class="p">(</span><span class="n">target_dt</span><span class="p">,</span> <span class="n">parts</span><span class="p">,</span> <span class="n">before_target</span><span class="o">=</span><span class="n">before</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">closest_ds</span><span class="o">.</span><span class="n">isoformat</span><span class="p">()</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[19/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/slack_operator.html
----------------------------------------------------------------------
diff --git a/_modules/slack_operator.html b/_modules/slack_operator.html
new file mode 100644
index 0000000..4076627
--- /dev/null
+++ b/_modules/slack_operator.html
@@ -0,0 +1,304 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>slack_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>slack_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for slack_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">slackclient</span> <span class="kn">import</span> <span class="n">SlackClient</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+<span class="kn">import</span> <span class="nn">json</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+
+
+<div class="viewcode-block" id="SlackAPIOperator"><a class="viewcode-back" href="../code.html#airflow.operators.SlackAPIOperator">[docs]</a><span class="k">class</span> <span class="nc">SlackAPIOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Base Slack Operator</span>
+<span class="sd"> The SlackAPIPostOperator is derived from this operator.</span>
+<span class="sd"> In the future additional Slack API Operators will be derived from this class as well</span>
+
+<span class="sd"> :param token: Slack API token (https://api.slack.com/web)</span>
+<span class="sd"> :type token: string</span>
+<span class="sd"> :param method: The Slack API Method to Call (https://api.slack.com/methods)</span>
+<span class="sd"> :type method: string</span>
+<span class="sd"> :param api_params: API Method call parameters (https://api.slack.com/methods)</span>
+<span class="sd"> :type api_params: dict</span>
+<span class="sd"> """</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">token</span><span class="o">=</span><span class="s1">'unset'</span><span class="p">,</span>
+ <span class="n">method</span><span class="o">=</span><span class="s1">'unset'</span><span class="p">,</span>
+ <span class="n">api_params</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SlackAPIOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">token</span> <span class="o">=</span> <span class="n">token</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">=</span> <span class="n">method</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">api_params</span> <span class="o">=</span> <span class="n">api_params</span>
+
+<div class="viewcode-block" id="SlackAPIOperator.construct_api_call_params"><a class="viewcode-back" href="../code.html#airflow.operators.SlackAPIOperator.construct_api_call_params">[docs]</a> <span class="k">def</span> <span class="nf">construct_api_call_params</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Used by the execute function. Allows templating on the source fields of the api_call_params dict before construction</span>
+
+<span class="sd"> Override in child classes.</span>
+<span class="sd"> Each SlackAPIOperator child class is responsible for having a construct_api_call_params function</span>
+<span class="sd"> which sets self.api_call_params with a dict of API call parameters (https://api.slack.com/methods)</span>
+<span class="sd"> """</span>
+
+ <span class="k">pass</span></div>
+
+<div class="viewcode-block" id="SlackAPIOperator.execute"><a class="viewcode-back" href="../code.html#airflow.operators.SlackAPIOperator.execute">[docs]</a> <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> SlackAPIOperator calls will not fail even if the call is not unsuccessful.</span>
+<span class="sd"> It should not prevent a DAG from completing in success</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">api_params</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">construct_api_call_params</span><span class="p">()</span>
+ <span class="n">sc</span> <span class="o">=</span> <span class="n">SlackClient</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">token</span><span class="p">)</span>
+ <span class="n">rc</span> <span class="o">=</span> <span class="n">sc</span><span class="o">.</span><span class="n">api_call</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">method</span><span class="p">,</span> <span class="o">**</span><span class="bp">self</span><span class="o">.</span><span class="n">api_params</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">rc</span><span class="p">[</span><span class="s1">'ok'</span><span class="p">]:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Slack API call failed ({})"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">rc</span><span class="p">[</span><span class="s1">'error'</span><span class="p">]))</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Slack API call failed: ({})"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">rc</span><span class="p">[</span><span class="s1">'error'</span><span class="p">]))</span></div></div>
+
+
+<div class="viewcode-block" id="SlackAPIPostOperator"><a class="viewcode-back" href="../code.html#airflow.operators.SlackAPIPostOperator">[docs]</a><span class="k">class</span> <span class="nc">SlackAPIPostOperator</span><span class="p">(</span><span class="n">SlackAPIOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Posts messages to a slack channel</span>
+
+<span class="sd"> :param channel: channel in which to post message on slack name (#general) or ID (C12318391)</span>
+<span class="sd"> :type channel: string</span>
+<span class="sd"> :param username: Username that airflow will be posting to Slack as</span>
+<span class="sd"> :type username: string</span>
+<span class="sd"> :param text: message to send to slack</span>
+<span class="sd"> :type text: string</span>
+<span class="sd"> :param icon_url: url to icon used for this message</span>
+<span class="sd"> :type icon_url: string</span>
+<span class="sd"> :param attachments: extra formatting details - see https://api.slack.com/docs/attachments</span>
+<span class="sd"> :type attachments: array of hashes</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'username'</span><span class="p">,</span> <span class="s1">'text'</span><span class="p">,</span> <span class="s1">'attachments'</span><span class="p">)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#FFBA40'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">channel</span><span class="o">=</span><span class="s1">'#general'</span><span class="p">,</span>
+ <span class="n">username</span><span class="o">=</span><span class="s1">'Airflow'</span><span class="p">,</span>
+ <span class="n">text</span><span class="o">=</span><span class="s1">'No message has been set.</span><span class="se">\n</span><span class="s1">'</span>
+ <span class="s1">'Here is a cat video instead</span><span class="se">\n</span><span class="s1">'</span>
+ <span class="s1">'https://www.youtube.com/watch?v=J---aiyznGQ'</span><span class="p">,</span>
+ <span class="n">icon_url</span><span class="o">=</span><span class="s1">'https://raw.githubusercontent.com/airbnb/airflow/master/airflow/www/static/pin_100.png'</span><span class="p">,</span>
+ <span class="n">attachments</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">=</span> <span class="s1">'chat.postMessage'</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">channel</span> <span class="o">=</span> <span class="n">channel</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">username</span> <span class="o">=</span> <span class="n">username</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">text</span> <span class="o">=</span> <span class="n">text</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">icon_url</span> <span class="o">=</span> <span class="n">icon_url</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">attachments</span> <span class="o">=</span> <span class="n">attachments</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SlackAPIPostOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">method</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">method</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">construct_api_call_params</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">api_params</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'channel'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">channel</span><span class="p">,</span>
+ <span class="s1">'username'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">username</span><span class="p">,</span>
+ <span class="s1">'text'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">text</span><span class="p">,</span>
+ <span class="s1">'icon_url'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">icon_url</span><span class="p">,</span>
+ <span class="s1">'attachments'</span><span class="p">:</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">attachments</span><span class="p">),</span>
+ <span class="p">}</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/sqlite_hook.html
----------------------------------------------------------------------
diff --git a/_modules/sqlite_hook.html b/_modules/sqlite_hook.html
new file mode 100644
index 0000000..ebe6497
--- /dev/null
+++ b/_modules/sqlite_hook.html
@@ -0,0 +1,222 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>sqlite_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>sqlite_hook</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for sqlite_hook</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">sqlite3</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks.dbapi_hook</span> <span class="kn">import</span> <span class="n">DbApiHook</span>
+
+
+<div class="viewcode-block" id="SqliteHook"><a class="viewcode-back" href="../code.html#airflow.hooks.SqliteHook">[docs]</a><span class="k">class</span> <span class="nc">SqliteHook</span><span class="p">(</span><span class="n">DbApiHook</span><span class="p">):</span>
+
+ <span class="sd">"""</span>
+<span class="sd"> Interact with SQLite.</span>
+<span class="sd"> """</span>
+
+ <span class="n">conn_name_attr</span> <span class="o">=</span> <span class="s1">'sqlite_conn_id'</span>
+ <span class="n">default_conn_name</span> <span class="o">=</span> <span class="s1">'sqlite_default'</span>
+ <span class="n">supports_autocommit</span> <span class="o">=</span> <span class="bp">False</span>
+
+<div class="viewcode-block" id="SqliteHook.get_conn"><a class="viewcode-back" href="../code.html#airflow.hooks.SqliteHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a sqlite connection object</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sqlite_conn_id</span><span class="p">)</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="n">sqlite3</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">host</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">conn</span></div></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/ssh_execute_operator.html
----------------------------------------------------------------------
diff --git a/_modules/ssh_execute_operator.html b/_modules/ssh_execute_operator.html
new file mode 100644
index 0000000..c49e840
--- /dev/null
+++ b/_modules/ssh_execute_operator.html
@@ -0,0 +1,343 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>ssh_execute_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>ssh_execute_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for ssh_execute_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">bytes</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">import</span> <span class="nn">subprocess</span>
+<span class="kn">from</span> <span class="nn">subprocess</span> <span class="kn">import</span> <span class="n">STDOUT</span>
+
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+
+
+<span class="k">class</span> <span class="nc">SSHTempFileContent</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+ <span class="sd">"""This class prvides a functionality that creates tempfile</span>
+<span class="sd"> with given content at remote host.</span>
+<span class="sd"> Use like::</span>
+
+<span class="sd"> with SSHTempFileContent(ssh_hook, content) as tempfile:</span>
+<span class="sd"> ...</span>
+
+<span class="sd"> In this case, a temporary file ``tempfile``</span>
+<span class="sd"> with content ``content`` is created where ``ssh_hook`` designate.</span>
+
+<span class="sd"> Note that this isn't safe because other processes</span>
+<span class="sd"> at remote host can read and write that tempfile.</span>
+
+<span class="sd"> :param ssh_hook: A SSHHook that indicates a remote host</span>
+<span class="sd"> where you want to create tempfile</span>
+<span class="sd"> :param content: Initial content of creating temprary file</span>
+<span class="sd"> :type content: string</span>
+<span class="sd"> :param prefix: The prefix string you want to use for the temprary file</span>
+<span class="sd"> :type prefix: string</span>
+<span class="sd"> """</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ssh_hook</span><span class="p">,</span> <span class="n">content</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="s2">"tmp"</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_ssh_hook</span> <span class="o">=</span> <span class="n">ssh_hook</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_content</span> <span class="o">=</span> <span class="n">content</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_prefix</span> <span class="o">=</span> <span class="n">prefix</span>
+
+ <span class="k">def</span> <span class="nf">__enter__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">ssh_hook</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_ssh_hook</span>
+ <span class="n">string</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_content</span>
+ <span class="n">prefix</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_prefix</span>
+
+ <span class="n">pmktemp</span> <span class="o">=</span> <span class="n">ssh_hook</span><span class="o">.</span><span class="n">Popen</span><span class="p">([</span><span class="s2">"-q"</span><span class="p">,</span>
+ <span class="s2">"mktemp"</span><span class="p">,</span> <span class="s2">"--tmpdir"</span><span class="p">,</span> <span class="n">prefix</span> <span class="o">+</span> <span class="s2">"_XXXXXX"</span><span class="p">],</span>
+ <span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span>
+ <span class="n">stderr</span><span class="o">=</span><span class="n">STDOUT</span><span class="p">)</span>
+ <span class="n">tempfile</span> <span class="o">=</span> <span class="n">pmktemp</span><span class="o">.</span><span class="n">communicate</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">rstrip</span><span class="p">()</span>
+ <span class="n">pmktemp</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">pmktemp</span><span class="o">.</span><span class="n">returncode</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Failed to create remote temp file"</span><span class="p">)</span>
+
+ <span class="n">ptee</span> <span class="o">=</span> <span class="n">ssh_hook</span><span class="o">.</span><span class="n">Popen</span><span class="p">([</span><span class="s2">"-q"</span><span class="p">,</span> <span class="s2">"tee"</span><span class="p">,</span> <span class="n">tempfile</span><span class="p">],</span>
+ <span class="n">stdin</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span>
+ <span class="c1"># discard stdout</span>
+ <span class="n">stderr</span><span class="o">=</span><span class="n">STDOUT</span><span class="p">)</span>
+ <span class="n">ptee</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="n">string</span><span class="p">,</span> <span class="s1">'utf_8'</span><span class="p">))</span>
+ <span class="n">ptee</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">ptee</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">ptee</span><span class="o">.</span><span class="n">returncode</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Failed to write to remote temp file"</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">_tempfile</span> <span class="o">=</span> <span class="n">tempfile</span>
+ <span class="k">return</span> <span class="n">tempfile</span>
+
+ <span class="k">def</span> <span class="nf">__exit__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">type</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">traceback</span><span class="p">):</span>
+ <span class="n">sp</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_ssh_hook</span><span class="o">.</span><span class="n">Popen</span><span class="p">([</span><span class="s2">"-q"</span><span class="p">,</span> <span class="s2">"rm"</span><span class="p">,</span> <span class="s2">"-f"</span><span class="p">,</span> <span class="s2">"--"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_tempfile</span><span class="p">])</span>
+ <span class="n">sp</span><span class="o">.</span><span class="n">communicate</span><span class="p">()</span>
+ <span class="n">sp</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">sp</span><span class="o">.</span><span class="n">returncode</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Failed to remove to remote temp file"</span><span class="p">)</span>
+ <span class="k">return</span> <span class="bp">False</span>
+
+
+<div class="viewcode-block" id="SSHExecuteOperator"><a class="viewcode-back" href="../code.html#airflow.contrib.operators.SSHExecuteOperator">[docs]</a><span class="k">class</span> <span class="nc">SSHExecuteOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Execute a Bash script, command or set of commands at remote host.</span>
+
+<span class="sd"> :param ssh_hook: A SSHHook that indicates the remote host</span>
+<span class="sd"> you want to run the script</span>
+<span class="sd"> :param ssh_hook: SSHHook</span>
+<span class="sd"> :param bash_command: The command, set of commands or reference to a</span>
+<span class="sd"> bash script (must be '.sh') to be executed.</span>
+<span class="sd"> :type bash_command: string</span>
+<span class="sd"> :param env: If env is not None, it must be a mapping that defines the</span>
+<span class="sd"> environment variables for the new process; these are used instead</span>
+<span class="sd"> of inheriting the current process environment, which is the default</span>
+<span class="sd"> behavior.</span>
+<span class="sd"> :type env: dict</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s2">"bash_command"</span><span class="p">,</span> <span class="s2">"env"</span><span class="p">,)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s2">".sh"</span><span class="p">,</span> <span class="s2">".bash"</span><span class="p">,)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">ssh_hook</span><span class="p">,</span>
+ <span class="n">bash_command</span><span class="p">,</span>
+ <span class="n">xcom_push</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">env</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SSHExecuteOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bash_command</span> <span class="o">=</span> <span class="n">bash_command</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">env</span> <span class="o">=</span> <span class="n">env</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">ssh_hook</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">xcom_push</span> <span class="o">=</span> <span class="n">xcom_push</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">bash_command</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">bash_command</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span>
+ <span class="n">host</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">_host_ref</span><span class="p">()</span>
+
+ <span class="k">with</span> <span class="n">SSHTempFileContent</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bash_command</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">task_id</span><span class="p">)</span> <span class="k">as</span> <span class="n">remote_file_path</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Temporary script "</span>
+ <span class="s2">"location : {0}:{1}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">host</span><span class="p">,</span> <span class="n">remote_file_path</span><span class="p">))</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running command: "</span> <span class="o">+</span> <span class="n">bash_command</span><span class="p">)</span>
+
+ <span class="n">sp</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span>
+ <span class="p">[</span><span class="s1">'-q'</span><span class="p">,</span> <span class="s1">'bash'</span><span class="p">,</span> <span class="n">remote_file_path</span><span class="p">],</span>
+ <span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span> <span class="n">stderr</span><span class="o">=</span><span class="n">STDOUT</span><span class="p">,</span>
+ <span class="n">env</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">env</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">sp</span> <span class="o">=</span> <span class="n">sp</span>
+
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Output:"</span><span class="p">)</span>
+ <span class="n">line</span> <span class="o">=</span> <span class="s1">''</span>
+ <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="nb">iter</span><span class="p">(</span><span class="n">sp</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">readline</span><span class="p">,</span> <span class="n">b</span><span class="s1">''</span><span class="p">):</span>
+ <span class="n">line</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
+ <span class="n">sp</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Command exited with "</span>
+ <span class="s2">"return code {0}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">sp</span><span class="o">.</span><span class="n">returncode</span><span class="p">))</span>
+ <span class="k">if</span> <span class="n">sp</span><span class="o">.</span><span class="n">returncode</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Bash command failed"</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">xcom_push</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">line</span>
+
+ <span class="k">def</span> <span class="nf">on_kill</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="c1"># TODO: Cleanup remote tempfile</span>
+ <span class="c1"># TODO: kill `mktemp` or `tee` too when they are alive.</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Sending SIGTERM signal to bash subprocess'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sp</span><span class="o">.</span><span class="n">terminate</span><span class="p">()</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/ssh_hook.html
----------------------------------------------------------------------
diff --git a/_modules/ssh_hook.html b/_modules/ssh_hook.html
new file mode 100644
index 0000000..8103f25
--- /dev/null
+++ b/_modules/ssh_hook.html
@@ -0,0 +1,353 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>ssh_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>ssh_hook</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for ssh_hook</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Copyright 2012-2015 Spotify AB</span>
+<span class="c1"># Ported to Airflow by Bolke de Bruin</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="c1">#</span>
+<span class="c1"># This is a port of Luigi's ssh implementation. All credits go there.</span>
+<span class="kn">import</span> <span class="nn">subprocess</span>
+<span class="kn">from</span> <span class="nn">contextlib</span> <span class="kn">import</span> <span class="n">contextmanager</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="kn">import</span> <span class="n">BaseHook</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+
+<span class="kn">import</span> <span class="nn">logging</span>
+
+
+<div class="viewcode-block" id="SSHHook"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.SSHHook">[docs]</a><span class="k">class</span> <span class="nc">SSHHook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Light-weight remote execution library and utilities.</span>
+
+<span class="sd"> Using this hook (which is just a convenience wrapper for subprocess),</span>
+<span class="sd"> is created to let you stream data from a remotely stored file.</span>
+
+<span class="sd"> As a bonus, :class:`SSHHook` also provides a really cool feature that let's you</span>
+<span class="sd"> set up ssh tunnels super easily using a python context manager (there is an example</span>
+<span class="sd"> in the integration part of unittests).</span>
+
+<span class="sd"> :param key_file: Typically the SSHHook uses the keys that are used by the user</span>
+<span class="sd"> airflow is running under. This sets the behavior to use another file instead.</span>
+<span class="sd"> :type key_file: str</span>
+<span class="sd"> :param connect_timeout: sets the connection timeout for this connection.</span>
+<span class="sd"> :type connect_timeout: int</span>
+<span class="sd"> :param no_host_key_check: whether to check to host key. If True host keys will not</span>
+<span class="sd"> be checked, but are also not stored in the current users's known_hosts file.</span>
+<span class="sd"> :type no_host_key_check: bool</span>
+<span class="sd"> :param tty: allocate a tty.</span>
+<span class="sd"> :type tty: bool</span>
+<span class="sd"> :param sshpass: Use to non-interactively perform password authentication by using</span>
+<span class="sd"> sshpass.</span>
+<span class="sd"> :type sshpass: bool</span>
+<span class="sd"> """</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">conn_id</span><span class="o">=</span><span class="s1">'ssh_default'</span><span class="p">):</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="n">conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">key_file</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'key_file'</span><span class="p">,</span> <span class="bp">None</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">connect_timeout</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'connect_timeout'</span><span class="p">,</span> <span class="bp">None</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">no_host_key_check</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'no_host_key_check'</span><span class="p">,</span> <span class="bp">False</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">tty</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'tty'</span><span class="p">,</span> <span class="bp">False</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sshpass</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'sshpass'</span><span class="p">,</span> <span class="bp">False</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="o">=</span> <span class="n">conn</span>
+
+ <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">pass</span>
+
+ <span class="k">def</span> <span class="nf">_host_ref</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">:</span>
+ <span class="k">return</span> <span class="s2">"{0}@{1}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">host</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">host</span>
+
+ <span class="k">def</span> <span class="nf">_prepare_command</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">cmd</span><span class="p">):</span>
+ <span class="n">connection_cmd</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"ssh"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">_host_ref</span><span class="p">(),</span> <span class="s2">"-o"</span><span class="p">,</span> <span class="s2">"ControlMaster=no"</span><span class="p">]</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">sshpass</span><span class="p">:</span>
+ <span class="n">connection_cmd</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"sshpass"</span><span class="p">,</span> <span class="s2">"-e"</span><span class="p">]</span> <span class="o">+</span> <span class="n">connection_cmd</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">connection_cmd</span> <span class="o">+=</span> <span class="p">[</span><span class="s2">"-o"</span><span class="p">,</span> <span class="s2">"BatchMode=yes"</span><span class="p">]</span> <span class="c1"># no password prompts</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">port</span><span class="p">:</span>
+ <span class="n">connection_cmd</span> <span class="o">+=</span> <span class="p">[</span><span class="s2">"-p"</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">port</span><span class="p">)]</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">connect_timeout</span><span class="p">:</span>
+ <span class="n">connection_cmd</span> <span class="o">+=</span> <span class="p">[</span><span class="s2">"-o"</span><span class="p">,</span> <span class="s2">"ConnectionTimeout={}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">connect_timeout</span><span class="p">)]</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">no_host_key_check</span><span class="p">:</span>
+ <span class="n">connection_cmd</span> <span class="o">+=</span> <span class="p">[</span><span class="s2">"-o"</span><span class="p">,</span> <span class="s2">"UserKnownHostsFile=/dev/null"</span><span class="p">,</span>
+ <span class="s2">"-o"</span><span class="p">,</span> <span class="s2">"StrictHostKeyChecking=no"</span><span class="p">]</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">key_file</span><span class="p">:</span>
+ <span class="n">connection_cmd</span> <span class="o">+=</span> <span class="p">[</span><span class="s2">"-i"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">key_file</span><span class="p">]</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">tty</span><span class="p">:</span>
+ <span class="n">connection_cmd</span> <span class="o">+=</span> <span class="p">[</span><span class="s2">"-t"</span><span class="p">]</span>
+
+ <span class="n">connection_cmd</span> <span class="o">+=</span> <span class="n">cmd</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"SSH cmd: {} "</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">connection_cmd</span><span class="p">))</span>
+
+ <span class="k">return</span> <span class="n">connection_cmd</span>
+
+<div class="viewcode-block" id="SSHHook.Popen"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.SSHHook.Popen">[docs]</a> <span class="k">def</span> <span class="nf">Popen</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">cmd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Remote Popen</span>
+
+<span class="sd"> :param cmd: command to remotely execute</span>
+<span class="sd"> :param kwargs: extra arguments to Popen (see subprocess.Popen)</span>
+<span class="sd"> :return: handle to subprocess</span>
+<span class="sd"> """</span>
+ <span class="n">prefixed_cmd</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_prepare_command</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span><span class="n">prefixed_cmd</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="SSHHook.check_output"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.SSHHook.check_output">[docs]</a> <span class="k">def</span> <span class="nf">check_output</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">cmd</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes a remote command and returns the stdout a remote process.</span>
+<span class="sd"> Simplified version of Popen when you only want the output as a string and detect any errors.</span>
+
+<span class="sd"> :param cmd: command to remotely execute</span>
+<span class="sd"> :return: stdout</span>
+<span class="sd"> """</span>
+ <span class="n">p</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span><span class="n">cmd</span><span class="p">,</span> <span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span> <span class="n">stderr</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">)</span>
+ <span class="n">output</span><span class="p">,</span> <span class="n">stderr</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="n">communicate</span><span class="p">()</span>
+
+ <span class="k">if</span> <span class="n">p</span><span class="o">.</span><span class="n">returncode</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
+ <span class="c1"># I like this better: RemoteCalledProcessError(p.returncode, cmd, self.host, output=output)</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Cannot execute {} on {}. Error code is: {}. Output: {}, Stderr: {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">cmd</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span><span class="o">.</span><span class="n">host</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">returncode</span><span class="p">,</span> <span class="n">output</span><span class="p">,</span> <span class="n">stderr</span><span class="p">))</span>
+
+ <span class="k">return</span> <span class="n">output</span></div>
+
+ <span class="nd">@contextmanager</span>
+<div class="viewcode-block" id="SSHHook.tunnel"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.SSHHook.tunnel">[docs]</a> <span class="k">def</span> <span class="nf">tunnel</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">local_port</span><span class="p">,</span> <span class="n">remote_port</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">remote_host</span><span class="o">=</span><span class="s2">"localhost"</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Creates a tunnel between two hosts. Like ssh -L <LOCAL_PORT>:host:<REMOTE_PORT>.</span>
+<span class="sd"> Remember to close() the returned "tunnel" object in order to clean up</span>
+<span class="sd"> after yourself when you are done with the tunnel.</span>
+
+<span class="sd"> :param local_port:</span>
+<span class="sd"> :type local_port: int</span>
+<span class="sd"> :param remote_port:</span>
+<span class="sd"> :type remote_port: int</span>
+<span class="sd"> :param remote_host:</span>
+<span class="sd"> :type remote_host: str</span>
+<span class="sd"> :return:</span>
+<span class="sd"> """</span>
+ <span class="n">tunnel_host</span> <span class="o">=</span> <span class="s2">"{0}:{1}:{2}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">local_port</span><span class="p">,</span> <span class="n">remote_host</span><span class="p">,</span> <span class="n">remote_port</span><span class="p">)</span>
+ <span class="n">proc</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">Popen</span><span class="p">([</span><span class="s2">"-L"</span><span class="p">,</span> <span class="n">tunnel_host</span><span class="p">,</span> <span class="s2">"echo -n ready && cat"</span><span class="p">],</span>
+ <span class="n">stdin</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span> <span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span>
+ <span class="p">)</span>
+
+ <span class="n">ready</span> <span class="o">=</span> <span class="n">proc</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
+ <span class="k">assert</span> <span class="n">ready</span> <span class="o">==</span> <span class="n">b</span><span class="s2">"ready"</span><span class="p">,</span> <span class="s2">"Did not get 'ready' from remote"</span>
+ <span class="k">yield</span>
+ <span class="n">proc</span><span class="o">.</span><span class="n">communicate</span><span class="p">()</span>
+ <span class="k">assert</span> <span class="n">proc</span><span class="o">.</span><span class="n">returncode</span> <span class="o">==</span> <span class="mi">0</span><span class="p">,</span> <span class="s2">"Tunnel process did unclean exit (returncode {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">proc</span><span class="o">.</span><span class="n">returncode</span><span class="p">)</span></div></div>
+
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/vertica_hook.html
----------------------------------------------------------------------
diff --git a/_modules/vertica_hook.html b/_modules/vertica_hook.html
new file mode 100644
index 0000000..ae7424f
--- /dev/null
+++ b/_modules/vertica_hook.html
@@ -0,0 +1,247 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>vertica_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>vertica_hook</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for vertica_hook</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="c1">#</span>
+
+<span class="kn">from</span> <span class="nn">vertica_python</span> <span class="kn">import</span> <span class="n">connect</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks.dbapi_hook</span> <span class="kn">import</span> <span class="n">DbApiHook</span>
+
+<div class="viewcode-block" id="VerticaHook"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.VerticaHook">[docs]</a><span class="k">class</span> <span class="nc">VerticaHook</span><span class="p">(</span><span class="n">DbApiHook</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> Interact with Vertica.</span>
+<span class="sd"> '''</span>
+
+ <span class="n">conn_name_attr</span> <span class="o">=</span> <span class="s1">'vertica_conn_id'</span>
+ <span class="n">default_conn_name</span> <span class="o">=</span> <span class="s1">'vertica_default'</span>
+ <span class="n">supports_autocommit</span> <span class="o">=</span> <span class="bp">True</span>
+
+<div class="viewcode-block" id="VerticaHook.get_conn"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.VerticaHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns verticaql connection object</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">vertica_conn_id</span><span class="p">)</span>
+ <span class="n">conn_config</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s2">"user"</span><span class="p">:</span> <span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">,</span>
+ <span class="s2">"password"</span><span class="p">:</span> <span class="n">conn</span><span class="o">.</span><span class="n">password</span> <span class="ow">or</span> <span class="s1">''</span><span class="p">,</span>
+ <span class="s2">"database"</span><span class="p">:</span> <span class="n">conn</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span>
+ <span class="p">}</span>
+
+ <span class="n">conn_config</span><span class="p">[</span><span class="s2">"host"</span><span class="p">]</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">host</span> <span class="ow">or</span> <span class="s1">'localhost'</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">conn</span><span class="o">.</span><span class="n">port</span><span class="p">:</span>
+ <span class="n">conn_config</span><span class="p">[</span><span class="s2">"port"</span><span class="p">]</span> <span class="o">=</span> <span class="mi">5433</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">conn_config</span><span class="p">[</span><span class="s2">"port"</span><span class="p">]</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">port</span><span class="p">)</span>
+
+ <span class="n">conn</span> <span class="o">=</span> <span class="n">connect</span><span class="p">(</span><span class="o">**</span><span class="n">conn_config</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">conn</span></div></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[24/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/hive_operator.html
----------------------------------------------------------------------
diff --git a/_modules/hive_operator.html b/_modules/hive_operator.html
new file mode 100644
index 0000000..6b6f4bb
--- /dev/null
+++ b/_modules/hive_operator.html
@@ -0,0 +1,272 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>hive_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>hive_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for hive_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">import</span> <span class="nn">re</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">HiveCliHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="HiveOperator"><a class="viewcode-back" href="../code.html#airflow.operators.HiveOperator">[docs]</a><span class="k">class</span> <span class="nc">HiveOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes hql code in a specific Hive database.</span>
+
+<span class="sd"> :param hql: the hql to be executed</span>
+<span class="sd"> :type hql: string</span>
+<span class="sd"> :param hive_cli_conn_id: reference to the Hive database</span>
+<span class="sd"> :type hive_cli_conn_id: string</span>
+<span class="sd"> :param hiveconf_jinja_translate: when True, hiveconf-type templating</span>
+<span class="sd"> ${var} gets translated into jinja-type templating {{ var }}. Note that</span>
+<span class="sd"> you may want to use this along with the</span>
+<span class="sd"> ``DAG(user_defined_macros=myargs)`` parameter. View the DAG</span>
+<span class="sd"> object documentation for more details.</span>
+<span class="sd"> :type hiveconf_jinja_translate: boolean</span>
+<span class="sd"> :param script_begin_tag: If defined, the operator will get rid of the</span>
+<span class="sd"> part of the script before the first occurrence of `script_begin_tag`</span>
+<span class="sd"> :type script_begin_tag: str</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'hql'</span><span class="p">,</span> <span class="s1">'schema'</span><span class="p">)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.hql'</span><span class="p">,</span> <span class="s1">'.sql'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#f0e4ec'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span>
+ <span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="s1">'hive_cli_default'</span><span class="p">,</span>
+ <span class="n">schema</span><span class="o">=</span><span class="s1">'default'</span><span class="p">,</span>
+ <span class="n">hiveconf_jinja_translate</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">script_begin_tag</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">run_as_owner</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+
+ <span class="nb">super</span><span class="p">(</span><span class="n">HiveOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hiveconf_jinja_translate</span> <span class="o">=</span> <span class="n">hiveconf_jinja_translate</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hql</span> <span class="o">=</span> <span class="n">hql</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="n">schema</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span> <span class="o">=</span> <span class="n">hive_cli_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">script_begin_tag</span> <span class="o">=</span> <span class="n">script_begin_tag</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">run_as</span> <span class="o">=</span> <span class="bp">None</span>
+ <span class="k">if</span> <span class="n">run_as_owner</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">run_as</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">dag</span><span class="o">.</span><span class="n">owner</span>
+
+ <span class="k">def</span> <span class="nf">get_hook</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">HiveCliHook</span><span class="p">(</span><span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span><span class="p">,</span> <span class="n">run_as</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">run_as</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">prepare_template</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">hiveconf_jinja_translate</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hql</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span>
+ <span class="s2">"(\$\{([ a-zA-Z0-9_]*)\})"</span><span class="p">,</span> <span class="s2">"{{ \g<2> }}"</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">hql</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">script_begin_tag</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">script_begin_tag</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">hql</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hql</span> <span class="o">=</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">hql</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">script_begin_tag</span><span class="p">)[</span><span class="mi">1</span><span class="p">:])</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing: '</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">hql</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="n">hql</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hql</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">dry_run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">test_hql</span><span class="p">(</span><span class="n">hql</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hql</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">on_kill</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">kill</span><span class="p">()</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/hive_to_druid.html
----------------------------------------------------------------------
diff --git a/_modules/hive_to_druid.html b/_modules/hive_to_druid.html
new file mode 100644
index 0000000..8dfe4e4
--- /dev/null
+++ b/_modules/hive_to_druid.html
@@ -0,0 +1,316 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>hive_to_druid — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>hive_to_druid</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for hive_to_druid</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">HiveCliHook</span><span class="p">,</span> <span class="n">DruidHook</span><span class="p">,</span> <span class="n">HiveMetastoreHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="HiveToDruidTransfer"><a class="viewcode-back" href="../code.html#airflow.operators.HiveToDruidTransfer">[docs]</a><span class="k">class</span> <span class="nc">HiveToDruidTransfer</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Moves data from Hive to Druid, [del]note that for now the data is loaded</span>
+<span class="sd"> into memory before being pushed to Druid, so this operator should</span>
+<span class="sd"> be used for smallish amount of data.[/del]</span>
+
+<span class="sd"> :param sql: SQL query to execute against the Druid database</span>
+<span class="sd"> :type sql: str</span>
+<span class="sd"> :param druid_datasource: the datasource you want to ingest into in druid</span>
+<span class="sd"> :type druid_datasource: str</span>
+<span class="sd"> :param ts_dim: the timestamp dimension</span>
+<span class="sd"> :type ts_dim: str</span>
+<span class="sd"> :param metric_spec: the metrics you want to define for your data</span>
+<span class="sd"> :type metric_spec: list</span>
+<span class="sd"> :param hive_cli_conn_id: the hive connection id</span>
+<span class="sd"> :type hive_cli_conn_id: str</span>
+<span class="sd"> :param druid_ingest_conn_id: the druid ingest connection id</span>
+<span class="sd"> :type druid_ingest_conn_id: str</span>
+<span class="sd"> :param metastore_conn_id: the metastore connection id</span>
+<span class="sd"> :type metastore_conn_id: str</span>
+<span class="sd"> :param hadoop_dependency_coordinates: list of coordinates to squeeze</span>
+<span class="sd"> int the ingest json</span>
+<span class="sd"> :type hadoop_dependency_coordinates: list of str</span>
+<span class="sd"> :param intervals: list of time intervals that defines segments, this</span>
+<span class="sd"> is passed as is to the json object</span>
+<span class="sd"> :type intervals: list</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,</span> <span class="s1">'intervals'</span><span class="p">)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,)</span>
+ <span class="c1">#ui_color = '#a0e08c'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">sql</span><span class="p">,</span>
+ <span class="n">druid_datasource</span><span class="p">,</span>
+ <span class="n">ts_dim</span><span class="p">,</span>
+ <span class="n">metric_spec</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="s1">'hive_cli_default'</span><span class="p">,</span>
+ <span class="n">druid_ingest_conn_id</span><span class="o">=</span><span class="s1">'druid_ingest_default'</span><span class="p">,</span>
+ <span class="n">metastore_conn_id</span><span class="o">=</span><span class="s1">'metastore_default'</span><span class="p">,</span>
+ <span class="n">hadoop_dependency_coordinates</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">intervals</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">num_shards</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span>
+ <span class="n">target_partition_size</span><span class="o">=-</span><span class="mi">1</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HiveToDruidTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">druid_datasource</span> <span class="o">=</span> <span class="n">druid_datasource</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">ts_dim</span> <span class="o">=</span> <span class="n">ts_dim</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">intervals</span> <span class="o">=</span> <span class="n">intervals</span> <span class="ow">or</span> <span class="p">[</span><span class="s1">'{{ ds }}/{{ tomorrow_ds }}'</span><span class="p">]</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">num_shards</span> <span class="o">=</span> <span class="n">num_shards</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">target_partition_size</span> <span class="o">=</span> <span class="n">target_partition_size</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metric_spec</span> <span class="o">=</span> <span class="n">metric_spec</span> <span class="ow">or</span> <span class="p">[{</span>
+ <span class="s2">"name"</span><span class="p">:</span> <span class="s2">"count"</span><span class="p">,</span>
+ <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"count"</span><span class="p">}]</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span> <span class="o">=</span> <span class="n">hive_cli_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hadoop_dependency_coordinates</span> <span class="o">=</span> <span class="n">hadoop_dependency_coordinates</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">druid_ingest_conn_id</span> <span class="o">=</span> <span class="n">druid_ingest_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metastore_conn_id</span> <span class="o">=</span> <span class="n">metastore_conn_id</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">hive</span> <span class="o">=</span> <span class="n">HiveCliHook</span><span class="p">(</span><span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Extracting data from Hive"</span><span class="p">)</span>
+ <span class="n">hive_table</span> <span class="o">=</span> <span class="s1">'druid.'</span> <span class="o">+</span> <span class="n">context</span><span class="p">[</span><span class="s1">'task_instance_key_str'</span><span class="p">]</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'.'</span><span class="p">,</span> <span class="s1">'_'</span><span class="p">)</span>
+ <span class="n">sql</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">strip</span><span class="p">(</span><span class="s1">';'</span><span class="p">)</span>
+ <span class="n">hql</span> <span class="o">=</span> <span class="s2">"""</span><span class="se">\</span>
+<span class="s2"> set mapred.output.compress=false;</span>
+<span class="s2"> set hive.exec.compress.output=false;</span>
+<span class="s2"> DROP TABLE IF EXISTS {hive_table};</span>
+<span class="s2"> CREATE TABLE {hive_table}</span>
+<span class="s2"> ROW FORMAT DELIMITED FIELDS TERMINATED BY '</span><span class="se">\t</span><span class="s2">'</span>
+<span class="s2"> STORED AS TEXTFILE</span>
+<span class="s2"> TBLPROPERTIES ('serialization.null.format' = '')</span>
+<span class="s2"> AS</span>
+<span class="s2"> {sql}</span>
+<span class="s2"> """</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running command:</span><span class="se">\n</span><span class="s2"> {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">hql</span><span class="p">))</span>
+ <span class="n">hive</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span>
+
+ <span class="n">m</span> <span class="o">=</span> <span class="n">HiveMetastoreHook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">metastore_conn_id</span><span class="p">)</span>
+ <span class="n">t</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">get_table</span><span class="p">(</span><span class="n">hive_table</span><span class="p">)</span>
+
+ <span class="n">columns</span> <span class="o">=</span> <span class="p">[</span><span class="n">col</span><span class="o">.</span><span class="n">name</span> <span class="k">for</span> <span class="n">col</span> <span class="ow">in</span> <span class="n">t</span><span class="o">.</span><span class="n">sd</span><span class="o">.</span><span class="n">cols</span><span class="p">]</span>
+
+ <span class="n">hdfs_uri</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">get_table</span><span class="p">(</span><span class="n">hive_table</span><span class="p">)</span><span class="o">.</span><span class="n">sd</span><span class="o">.</span><span class="n">location</span>
+ <span class="n">pos</span> <span class="o">=</span> <span class="n">hdfs_uri</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'/user'</span><span class="p">)</span>
+ <span class="n">static_path</span> <span class="o">=</span> <span class="n">hdfs_uri</span><span class="p">[</span><span class="n">pos</span><span class="p">:]</span>
+
+ <span class="n">schema</span><span class="p">,</span> <span class="n">table</span> <span class="o">=</span> <span class="n">hive_table</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span>
+
+ <span class="n">druid</span> <span class="o">=</span> <span class="n">DruidHook</span><span class="p">(</span><span class="n">druid_ingest_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">druid_ingest_conn_id</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Inserting rows into Druid"</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"HDFS path: "</span> <span class="o">+</span> <span class="n">static_path</span><span class="p">)</span>
+
+ <span class="n">druid</span><span class="o">.</span><span class="n">load_from_hdfs</span><span class="p">(</span>
+ <span class="n">datasource</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">druid_datasource</span><span class="p">,</span>
+ <span class="n">intervals</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">intervals</span><span class="p">,</span>
+ <span class="n">static_path</span><span class="o">=</span><span class="n">static_path</span><span class="p">,</span> <span class="n">ts_dim</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">ts_dim</span><span class="p">,</span>
+ <span class="n">columns</span><span class="o">=</span><span class="n">columns</span><span class="p">,</span> <span class="n">num_shards</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">num_shards</span><span class="p">,</span> <span class="n">target_partition_size</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">target_partition_size</span><span class="p">,</span>
+ <span class="n">metric_spec</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">metric_spec</span><span class="p">,</span> <span class="n">hadoop_dependency_coordinates</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hadoop_dependency_coordinates</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Load seems to have succeeded!"</span><span class="p">)</span>
+
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s2">"Cleaning up by dropping the temp "</span>
+ <span class="s2">"Hive table {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">hive_table</span><span class="p">))</span>
+ <span class="n">hql</span> <span class="o">=</span> <span class="s2">"DROP TABLE IF EXISTS {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">hive_table</span><span class="p">)</span>
+ <span class="n">hive</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/hive_to_mysql.html
----------------------------------------------------------------------
diff --git a/_modules/hive_to_mysql.html b/_modules/hive_to_mysql.html
new file mode 100644
index 0000000..13b223d
--- /dev/null
+++ b/_modules/hive_to_mysql.html
@@ -0,0 +1,294 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>hive_to_mysql — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>hive_to_mysql</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for hive_to_mysql</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">HiveServer2Hook</span><span class="p">,</span> <span class="n">MySqlHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+<span class="kn">from</span> <span class="nn">tempfile</span> <span class="kn">import</span> <span class="n">NamedTemporaryFile</span>
+
+<div class="viewcode-block" id="HiveToMySqlTransfer"><a class="viewcode-back" href="../code.html#airflow.operators.HiveToMySqlTransfer">[docs]</a><span class="k">class</span> <span class="nc">HiveToMySqlTransfer</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Moves data from Hive to MySQL, note that for now the data is loaded</span>
+<span class="sd"> into memory before being pushed to MySQL, so this operator should</span>
+<span class="sd"> be used for smallish amount of data.</span>
+
+<span class="sd"> :param sql: SQL query to execute against the MySQL database</span>
+<span class="sd"> :type sql: str</span>
+<span class="sd"> :param mysql_table: target MySQL table, use dot notation to target a</span>
+<span class="sd"> specific database</span>
+<span class="sd"> :type mysql_table: str</span>
+<span class="sd"> :param mysql_conn_id: source mysql connection</span>
+<span class="sd"> :type mysql_conn_id: str</span>
+<span class="sd"> :param hiveserver2_conn_id: destination hive connection</span>
+<span class="sd"> :type hiveserver2_conn_id: str</span>
+<span class="sd"> :param mysql_preoperator: sql statement to run against mysql prior to</span>
+<span class="sd"> import, typically use to truncate of delete in place of the data</span>
+<span class="sd"> coming in, allowing the task to be idempotent (running the task</span>
+<span class="sd"> twice won't double load data)</span>
+<span class="sd"> :type mysql_preoperator: str</span>
+<span class="sd"> :param mysql_postoperator: sql statement to run against mysql after the</span>
+<span class="sd"> import, typically used to move data from staging to production</span>
+<span class="sd"> and issue cleanup commands.</span>
+<span class="sd"> :type mysql_postoperator: str</span>
+<span class="sd"> :param bulk_load: flag to use bulk_load option. This loads mysql directly</span>
+<span class="sd"> from a tab-delimited text file using the LOAD DATA LOCAL INFILE command.</span>
+<span class="sd"> This option requires an extra connection parameter for the</span>
+<span class="sd"> destination MySQL connection: {'local_infile': true}.</span>
+<span class="sd"> :type bulk_load: bool</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,</span> <span class="s1">'mysql_table'</span><span class="p">,</span> <span class="s1">'mysql_preoperator'</span><span class="p">,</span>
+ <span class="s1">'mysql_postoperator'</span><span class="p">)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#a0e08c'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">sql</span><span class="p">,</span>
+ <span class="n">mysql_table</span><span class="p">,</span>
+ <span class="n">hiveserver2_conn_id</span><span class="o">=</span><span class="s1">'hiveserver2_default'</span><span class="p">,</span>
+ <span class="n">mysql_conn_id</span><span class="o">=</span><span class="s1">'mysql_default'</span><span class="p">,</span>
+ <span class="n">mysql_preoperator</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">mysql_postoperator</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">bulk_load</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HiveToMySqlTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">mysql_table</span> <span class="o">=</span> <span class="n">mysql_table</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">mysql_conn_id</span> <span class="o">=</span> <span class="n">mysql_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">mysql_preoperator</span> <span class="o">=</span> <span class="n">mysql_preoperator</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">mysql_postoperator</span> <span class="o">=</span> <span class="n">mysql_postoperator</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hiveserver2_conn_id</span> <span class="o">=</span> <span class="n">hiveserver2_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bulk_load</span> <span class="o">=</span> <span class="n">bulk_load</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">hive</span> <span class="o">=</span> <span class="n">HiveServer2Hook</span><span class="p">(</span><span class="n">hiveserver2_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hiveserver2_conn_id</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Extracting data from Hive"</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">bulk_load</span><span class="p">:</span>
+ <span class="n">tmpfile</span> <span class="o">=</span> <span class="n">NamedTemporaryFile</span><span class="p">()</span>
+ <span class="n">hive</span><span class="o">.</span><span class="n">to_csv</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">,</span> <span class="n">tmpfile</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">'</span><span class="se">\t</span><span class="s1">'</span><span class="p">,</span>
+ <span class="n">lineterminator</span><span class="o">=</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">,</span> <span class="n">output_header</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">results</span> <span class="o">=</span> <span class="n">hive</span><span class="o">.</span><span class="n">get_records</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+
+ <span class="n">mysql</span> <span class="o">=</span> <span class="n">MySqlHook</span><span class="p">(</span><span class="n">mysql_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_conn_id</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">mysql_preoperator</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running MySQL preoperator"</span><span class="p">)</span>
+ <span class="n">mysql</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_preoperator</span><span class="p">)</span>
+
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Inserting rows into MySQL"</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">bulk_load</span><span class="p">:</span>
+ <span class="n">mysql</span><span class="o">.</span><span class="n">bulk_load</span><span class="p">(</span><span class="n">table</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_table</span><span class="p">,</span> <span class="n">tmp_file</span><span class="o">=</span><span class="n">tmpfile</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
+ <span class="n">tmpfile</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">mysql</span><span class="o">.</span><span class="n">insert_rows</span><span class="p">(</span><span class="n">table</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_table</span><span class="p">,</span> <span class="n">rows</span><span class="o">=</span><span class="n">results</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">mysql_postoperator</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running MySQL postoperator"</span><span class="p">)</span>
+ <span class="n">mysql</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_postoperator</span><span class="p">)</span>
+
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Done."</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/hive_to_samba_operator.html
----------------------------------------------------------------------
diff --git a/_modules/hive_to_samba_operator.html b/_modules/hive_to_samba_operator.html
new file mode 100644
index 0000000..9eb09b5
--- /dev/null
+++ b/_modules/hive_to_samba_operator.html
@@ -0,0 +1,246 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>hive_to_samba_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>hive_to_samba_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for hive_to_samba_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">import</span> <span class="nn">tempfile</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">HiveServer2Hook</span><span class="p">,</span> <span class="n">SambaHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="Hive2SambaOperator"><a class="viewcode-back" href="../code.html#airflow.operators.Hive2SambaOperator">[docs]</a><span class="k">class</span> <span class="nc">Hive2SambaOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes hql code in a specific Hive database and loads the</span>
+<span class="sd"> results of the query as a csv to a Samba location.</span>
+
+<span class="sd"> :param hql: the hql to be exported</span>
+<span class="sd"> :type hql: string</span>
+<span class="sd"> :param hiveserver2_conn_id: reference to the hiveserver2 service</span>
+<span class="sd"> :type hiveserver2_conn_id: string</span>
+<span class="sd"> :param samba_conn_id: reference to the samba destination</span>
+<span class="sd"> :type samba_conn_id: string</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'hql'</span><span class="p">,</span> <span class="s1">'destination_filepath'</span><span class="p">)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.hql'</span><span class="p">,</span> <span class="s1">'.sql'</span><span class="p">,)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span>
+ <span class="n">destination_filepath</span><span class="p">,</span>
+ <span class="n">samba_conn_id</span><span class="o">=</span><span class="s1">'samba_default'</span><span class="p">,</span>
+ <span class="n">hiveserver2_conn_id</span><span class="o">=</span><span class="s1">'hiveserver2_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">Hive2SambaOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">hiveserver2_conn_id</span> <span class="o">=</span> <span class="n">hiveserver2_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">samba_conn_id</span> <span class="o">=</span> <span class="n">samba_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">destination_filepath</span> <span class="o">=</span> <span class="n">destination_filepath</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hql</span> <span class="o">=</span> <span class="n">hql</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">rstrip</span><span class="p">(</span><span class="s1">';'</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">samba</span> <span class="o">=</span> <span class="n">SambaHook</span><span class="p">(</span><span class="n">samba_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">samba_conn_id</span><span class="p">)</span>
+ <span class="n">hive</span> <span class="o">=</span> <span class="n">HiveServer2Hook</span><span class="p">(</span><span class="n">hiveserver2_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hiveserver2_conn_id</span><span class="p">)</span>
+ <span class="n">tmpfile</span> <span class="o">=</span> <span class="n">tempfile</span><span class="o">.</span><span class="n">NamedTemporaryFile</span><span class="p">()</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Fetching file from Hive"</span><span class="p">)</span>
+ <span class="n">hive</span><span class="o">.</span><span class="n">to_csv</span><span class="p">(</span><span class="n">hql</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hql</span><span class="p">,</span> <span class="n">csv_filepath</span><span class="o">=</span><span class="n">tmpfile</span><span class="o">.</span><span class="n">name</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Pushing to samba"</span><span class="p">)</span>
+ <span class="n">samba</span><span class="o">.</span><span class="n">push_from_local</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">destination_filepath</span><span class="p">,</span> <span class="n">tmpfile</span><span class="o">.</span><span class="n">name</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/http_hook.html
----------------------------------------------------------------------
diff --git a/_modules/http_hook.html b/_modules/http_hook.html
new file mode 100644
index 0000000..6f79193
--- /dev/null
+++ b/_modules/http_hook.html
@@ -0,0 +1,310 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>http_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>http_hook</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for http_hook</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="c1">#</span>
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">str</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">import</span> <span class="nn">requests</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="kn">import</span> <span class="n">BaseHook</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+
+
+<div class="viewcode-block" id="HttpHook"><a class="viewcode-back" href="../code.html#airflow.hooks.HttpHook">[docs]</a><span class="k">class</span> <span class="nc">HttpHook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Interact with HTTP servers.</span>
+<span class="sd"> """</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">method</span><span class="o">=</span><span class="s1">'POST'</span><span class="p">,</span> <span class="n">http_conn_id</span><span class="o">=</span><span class="s1">'http_default'</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">http_conn_id</span> <span class="o">=</span> <span class="n">http_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">=</span> <span class="n">method</span>
+
+ <span class="c1"># headers is required to make it required</span>
+<div class="viewcode-block" id="HttpHook.get_conn"><a class="viewcode-back" href="../code.html#airflow.hooks.HttpHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">headers</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns http session for use with requests</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">http_conn_id</span><span class="p">)</span>
+ <span class="n">session</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">host</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'http'</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span> <span class="o">=</span> <span class="s1">'http://'</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span>
+
+ <span class="k">if</span> <span class="n">conn</span><span class="o">.</span><span class="n">port</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span> <span class="o">+</span> <span class="s2">":"</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">port</span><span class="p">)</span> <span class="o">+</span> <span class="s2">"/"</span>
+ <span class="k">if</span> <span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">:</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">auth</span> <span class="o">=</span> <span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">,</span> <span class="n">conn</span><span class="o">.</span><span class="n">password</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">headers</span><span class="p">:</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">headers</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">headers</span><span class="p">)</span>
+
+ <span class="k">return</span> <span class="n">session</span></div>
+
+<div class="viewcode-block" id="HttpHook.run"><a class="viewcode-back" href="../code.html#airflow.hooks.HttpHook.run">[docs]</a> <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">endpoint</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">extra_options</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Performs the request</span>
+<span class="sd"> """</span>
+ <span class="n">extra_options</span> <span class="o">=</span> <span class="n">extra_options</span> <span class="ow">or</span> <span class="p">{}</span>
+
+ <span class="n">session</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">(</span><span class="n">headers</span><span class="p">)</span>
+
+ <span class="n">url</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span> <span class="o">+</span> <span class="n">endpoint</span>
+ <span class="n">req</span> <span class="o">=</span> <span class="bp">None</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">==</span> <span class="s1">'GET'</span><span class="p">:</span>
+ <span class="c1"># GET uses params</span>
+ <span class="n">req</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">method</span><span class="p">,</span>
+ <span class="n">url</span><span class="p">,</span>
+ <span class="n">params</span><span class="o">=</span><span class="n">data</span><span class="p">,</span>
+ <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="c1"># Others use data</span>
+ <span class="n">req</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">method</span><span class="p">,</span>
+ <span class="n">url</span><span class="p">,</span>
+ <span class="n">data</span><span class="o">=</span><span class="n">data</span><span class="p">,</span>
+ <span class="n">headers</span><span class="o">=</span><span class="n">headers</span><span class="p">)</span>
+
+ <span class="n">prepped_request</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">prepare_request</span><span class="p">(</span><span class="n">req</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Sending '"</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">+</span> <span class="s2">"' to url: "</span> <span class="o">+</span> <span class="n">url</span><span class="p">)</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_and_check</span><span class="p">(</span><span class="n">session</span><span class="p">,</span> <span class="n">prepped_request</span><span class="p">,</span> <span class="n">extra_options</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="HttpHook.run_and_check"><a class="viewcode-back" href="../code.html#airflow.hooks.HttpHook.run_and_check">[docs]</a> <span class="k">def</span> <span class="nf">run_and_check</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">session</span><span class="p">,</span> <span class="n">prepped_request</span><span class="p">,</span> <span class="n">extra_options</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Grabs extra options like timeout and actually runs the request,</span>
+<span class="sd"> checking for the result</span>
+<span class="sd"> """</span>
+ <span class="n">extra_options</span> <span class="o">=</span> <span class="n">extra_options</span> <span class="ow">or</span> <span class="p">{}</span>
+
+ <span class="n">response</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">send</span><span class="p">(</span>
+ <span class="n">prepped_request</span><span class="p">,</span>
+ <span class="n">stream</span><span class="o">=</span><span class="n">extra_options</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"stream"</span><span class="p">,</span> <span class="bp">False</span><span class="p">),</span>
+ <span class="n">verify</span><span class="o">=</span><span class="n">extra_options</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"verify"</span><span class="p">,</span> <span class="bp">False</span><span class="p">),</span>
+ <span class="n">proxies</span><span class="o">=</span><span class="n">extra_options</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"proxies"</span><span class="p">,</span> <span class="p">{}),</span>
+ <span class="n">cert</span><span class="o">=</span><span class="n">extra_options</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"cert"</span><span class="p">),</span>
+ <span class="n">timeout</span><span class="o">=</span><span class="n">extra_options</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"timeout"</span><span class="p">),</span>
+ <span class="n">allow_redirects</span><span class="o">=</span><span class="n">extra_options</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"allow_redirects"</span><span class="p">,</span> <span class="bp">True</span><span class="p">))</span>
+
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">response</span><span class="o">.</span><span class="n">raise_for_status</span><span class="p">()</span>
+ <span class="k">except</span> <span class="n">requests</span><span class="o">.</span><span class="n">exceptions</span><span class="o">.</span><span class="n">HTTPError</span><span class="p">:</span>
+ <span class="c1"># Tried rewrapping, but not supported. This way, it's possible</span>
+ <span class="c1"># to get reason and code for failure by checking first 3 chars</span>
+ <span class="c1"># for the code, or do a split on ':'</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"HTTP error: "</span> <span class="o">+</span> <span class="n">response</span><span class="o">.</span><span class="n">reason</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">!=</span> <span class="s1">'GET'</span><span class="p">:</span>
+ <span class="c1"># The sensor uses GET, so this prevents filling up the log</span>
+ <span class="c1"># with the body every time the GET 'misses'.</span>
+ <span class="c1"># That's ok to do, because GETs should be repeatable and</span>
+ <span class="c1"># all data should be visible in the log (no post data)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">status_code</span><span class="p">)</span><span class="o">+</span><span class="s2">":"</span><span class="o">+</span><span class="n">response</span><span class="o">.</span><span class="n">reason</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">response</span></div></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[18/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/vertica_operator.html
----------------------------------------------------------------------
diff --git a/_modules/vertica_operator.html b/_modules/vertica_operator.html
new file mode 100644
index 0000000..54caf66
--- /dev/null
+++ b/_modules/vertica_operator.html
@@ -0,0 +1,233 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>vertica_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>vertica_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for vertica_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks</span> <span class="kn">import</span> <span class="n">VerticaHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="VerticaOperator"><a class="viewcode-back" href="../code.html#airflow.contrib.operators.VerticaOperator">[docs]</a><span class="k">class</span> <span class="nc">VerticaOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes sql code in a specific Vertica database</span>
+
+<span class="sd"> :param vertica_conn_id: reference to a specific Vertica database</span>
+<span class="sd"> :type vertica_conn_id: string</span>
+<span class="sd"> :param sql: the sql code to be executed</span>
+<span class="sd"> :type sql: Can receive a str representing a sql statement,</span>
+<span class="sd"> a list of str (sql statements), or reference to a template file.</span>
+<span class="sd"> Template reference are recognized by str ending in '.sql'</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#b4e0ff'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="n">vertica_conn_id</span><span class="o">=</span><span class="s1">'vertica_default'</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">VerticaOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">vertica_conn_id</span> <span class="o">=</span> <span class="n">vertica_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing: '</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">))</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">VerticaHook</span><span class="p">(</span><span class="n">vertica_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">vertica_conn_id</span><span class="p">)</span>
+ <span class="n">hook</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/vertica_to_hive.html
----------------------------------------------------------------------
diff --git a/_modules/vertica_to_hive.html b/_modules/vertica_to_hive.html
new file mode 100644
index 0000000..abe189b
--- /dev/null
+++ b/_modules/vertica_to_hive.html
@@ -0,0 +1,316 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>vertica_to_hive — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>vertica_to_hive</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for vertica_to_hive</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">chr</span>
+<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">OrderedDict</span>
+<span class="kn">import</span> <span class="nn">unicodecsv</span> <span class="kn">as</span> <span class="nn">csv</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">from</span> <span class="nn">tempfile</span> <span class="kn">import</span> <span class="n">NamedTemporaryFile</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">HiveCliHook</span>
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks</span> <span class="kn">import</span> <span class="n">VerticaHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+<div class="viewcode-block" id="VerticaToHiveTransfer"><a class="viewcode-back" href="../code.html#airflow.contrib.operators.VerticaToHiveTransfer">[docs]</a><span class="k">class</span> <span class="nc">VerticaToHiveTransfer</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Moves data from Vertia to Hive. The operator runs</span>
+<span class="sd"> your query against Vertia, stores the file locally</span>
+<span class="sd"> before loading it into a Hive table. If the ``create`` or</span>
+<span class="sd"> ``recreate`` arguments are set to ``True``,</span>
+<span class="sd"> a ``CREATE TABLE`` and ``DROP TABLE`` statements are generated.</span>
+<span class="sd"> Hive data types are inferred from the cursor's metadata.</span>
+<span class="sd"> Note that the table generated in Hive uses ``STORED AS textfile``</span>
+<span class="sd"> which isn't the most efficient serialization format. If a</span>
+<span class="sd"> large amount of data is loaded and/or if the table gets</span>
+<span class="sd"> queried considerably, you may want to use this operator only to</span>
+<span class="sd"> stage the data into a temporary table before loading it into its</span>
+<span class="sd"> final destination using a ``HiveOperator``.</span>
+
+<span class="sd"> :param sql: SQL query to execute against the Vertia database</span>
+<span class="sd"> :type sql: str</span>
+<span class="sd"> :param hive_table: target Hive table, use dot notation to target a</span>
+<span class="sd"> specific database</span>
+<span class="sd"> :type hive_table: str</span>
+<span class="sd"> :param create: whether to create the table if it doesn't exist</span>
+<span class="sd"> :type create: bool</span>
+<span class="sd"> :param recreate: whether to drop and recreate the table at every execution</span>
+<span class="sd"> :type recreate: bool</span>
+<span class="sd"> :param partition: target partition as a dict of partition columns and values</span>
+<span class="sd"> :type partition: dict</span>
+<span class="sd"> :param delimiter: field delimiter in the file</span>
+<span class="sd"> :type delimiter: str</span>
+<span class="sd"> :param vertica_conn_id: source Vertica connection</span>
+<span class="sd"> :type vertica_conn_id: str</span>
+<span class="sd"> :param hive_conn_id: destination hive connection</span>
+<span class="sd"> :type hive_conn_id: str</span>
+
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,</span> <span class="s1">'partition'</span><span class="p">,</span> <span class="s1">'hive_table'</span><span class="p">)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#b4e0ff'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">sql</span><span class="p">,</span>
+ <span class="n">hive_table</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="nb">chr</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span>
+ <span class="n">vertica_conn_id</span><span class="o">=</span><span class="s1">'vertica_default'</span><span class="p">,</span>
+ <span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="s1">'hive_cli_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">VerticaToHiveTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span> <span class="o">=</span> <span class="n">hive_table</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">partition</span> <span class="o">=</span> <span class="n">partition</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">create</span> <span class="o">=</span> <span class="n">create</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">recreate</span> <span class="o">=</span> <span class="n">recreate</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">delimiter</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">vertica_conn_id</span> <span class="o">=</span> <span class="n">vertica_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span> <span class="o">=</span> <span class="n">hive_cli_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">partition</span> <span class="o">=</span> <span class="n">partition</span> <span class="ow">or</span> <span class="p">{}</span>
+
+ <span class="nd">@classmethod</span>
+ <span class="k">def</span> <span class="nf">type_map</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">vertica_type</span><span class="p">):</span>
+ <span class="c1"># vertica-python datatype.py donot provied the full type mapping access.</span>
+ <span class="c1"># Manual hack. Reference: https://github.com/uber/vertica-python/blob/master/vertica_python/vertica/column.py</span>
+ <span class="n">d</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="mi">5</span><span class="p">:</span> <span class="s1">'BOOLEAN'</span><span class="p">,</span>
+ <span class="mi">6</span><span class="p">:</span> <span class="s1">'INT'</span><span class="p">,</span>
+ <span class="mi">7</span><span class="p">:</span> <span class="s1">'FLOAT'</span><span class="p">,</span>
+ <span class="mi">8</span><span class="p">:</span> <span class="s1">'STRING'</span><span class="p">,</span>
+ <span class="mi">9</span><span class="p">:</span> <span class="s1">'STRING'</span><span class="p">,</span>
+ <span class="mi">16</span><span class="p">:</span> <span class="s1">'FLOAT'</span><span class="p">,</span>
+ <span class="p">}</span>
+ <span class="k">return</span> <span class="n">d</span><span class="p">[</span><span class="n">vertica_type</span><span class="p">]</span> <span class="k">if</span> <span class="n">vertica_type</span> <span class="ow">in</span> <span class="n">d</span> <span class="k">else</span> <span class="s1">'STRING'</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">hive</span> <span class="o">=</span> <span class="n">HiveCliHook</span><span class="p">(</span><span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span><span class="p">)</span>
+ <span class="n">vertica</span> <span class="o">=</span> <span class="n">VerticaHook</span><span class="p">(</span><span class="n">vertica_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">vertica_conn_id</span><span class="p">)</span>
+
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Dumping Vertica query results to local file"</span><span class="p">)</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="n">vertica</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">cursor</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
+ <span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+ <span class="k">with</span> <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="s2">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+ <span class="n">csv_writer</span> <span class="o">=</span> <span class="n">csv</span><span class="o">.</span><span class="n">writer</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span>
+ <span class="n">field_dict</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">()</span>
+ <span class="n">col_count</span> <span class="o">=</span> <span class="mi">0</span>
+ <span class="k">for</span> <span class="n">field</span> <span class="ow">in</span> <span class="n">cursor</span><span class="o">.</span><span class="n">description</span><span class="p">:</span>
+ <span class="n">col_count</span> <span class="o">+=</span> <span class="mi">1</span>
+ <span class="n">col_position</span> <span class="o">=</span> <span class="s2">"Column{position}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">position</span><span class="o">=</span><span class="n">col_count</span><span class="p">)</span>
+ <span class="n">field_dict</span><span class="p">[</span><span class="n">col_position</span> <span class="k">if</span> <span class="n">field</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="s1">''</span> <span class="k">else</span> <span class="n">field</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">type_map</span><span class="p">(</span><span class="n">field</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
+ <span class="n">csv_writer</span><span class="o">.</span><span class="n">writerows</span><span class="p">(</span><span class="n">cursor</span><span class="o">.</span><span class="n">iterate</span><span class="p">())</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
+ <span class="n">cursor</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Loading file into Hive"</span><span class="p">)</span>
+ <span class="n">hive</span><span class="o">.</span><span class="n">load_file</span><span class="p">(</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span><span class="p">,</span>
+ <span class="n">field_dict</span><span class="o">=</span><span class="n">field_dict</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">create</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">partition</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">recreate</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/webhdfs_hook.html
----------------------------------------------------------------------
diff --git a/_modules/webhdfs_hook.html b/_modules/webhdfs_hook.html
new file mode 100644
index 0000000..56be098
--- /dev/null
+++ b/_modules/webhdfs_hook.html
@@ -0,0 +1,287 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>webhdfs_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>webhdfs_hook</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for webhdfs_hook</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="kn">import</span> <span class="n">BaseHook</span>
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="kn">import</span> <span class="n">configuration</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">from</span> <span class="nn">hdfs</span> <span class="kn">import</span> <span class="n">InsecureClient</span><span class="p">,</span> <span class="n">HdfsError</span>
+
+<span class="n">_kerberos_security_mode</span> <span class="o">=</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"core"</span><span class="p">,</span> <span class="s2">"security"</span><span class="p">)</span> <span class="o">==</span> <span class="s2">"kerberos"</span>
+<span class="k">if</span> <span class="n">_kerberos_security_mode</span><span class="p">:</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="kn">from</span> <span class="nn">hdfs.ext.kerberos</span> <span class="kn">import</span> <span class="n">KerberosClient</span>
+ <span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Could not load the Kerberos extension for the WebHDFSHook."</span><span class="p">)</span>
+ <span class="k">raise</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+
+
+<span class="k">class</span> <span class="nc">AirflowWebHDFSHookException</span><span class="p">(</span><span class="n">AirflowException</span><span class="p">):</span>
+ <span class="k">pass</span>
+
+
+<div class="viewcode-block" id="WebHDFSHook"><a class="viewcode-back" href="../code.html#airflow.hooks.WebHDFSHook">[docs]</a><span class="k">class</span> <span class="nc">WebHDFSHook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Interact with HDFS. This class is a wrapper around the hdfscli library.</span>
+<span class="sd"> """</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">webhdfs_conn_id</span><span class="o">=</span><span class="s1">'webhdfs_default'</span><span class="p">,</span> <span class="n">proxy_user</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">webhdfs_conn_id</span> <span class="o">=</span> <span class="n">webhdfs_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">proxy_user</span> <span class="o">=</span> <span class="n">proxy_user</span>
+
+<div class="viewcode-block" id="WebHDFSHook.get_conn"><a class="viewcode-back" href="../code.html#airflow.hooks.WebHDFSHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a hdfscli InsecureClient object.</span>
+<span class="sd"> """</span>
+ <span class="n">nn_connections</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connections</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">webhdfs_conn_id</span><span class="p">)</span>
+ <span class="k">for</span> <span class="n">nn</span> <span class="ow">in</span> <span class="n">nn_connections</span><span class="p">:</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'Trying namenode {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">host</span><span class="p">))</span>
+ <span class="n">connection_str</span> <span class="o">=</span> <span class="s1">'http://{nn.host}:{nn.port}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">nn</span><span class="o">=</span><span class="n">nn</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">_kerberos_security_mode</span><span class="p">:</span>
+ <span class="n">client</span> <span class="o">=</span> <span class="n">KerberosClient</span><span class="p">(</span><span class="n">connection_str</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">proxy_user</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">proxy_user</span> <span class="ow">or</span> <span class="n">nn</span><span class="o">.</span><span class="n">login</span>
+ <span class="n">client</span> <span class="o">=</span> <span class="n">InsecureClient</span><span class="p">(</span><span class="n">connection_str</span><span class="p">,</span> <span class="n">user</span><span class="o">=</span><span class="n">proxy_user</span><span class="p">)</span>
+ <span class="n">client</span><span class="o">.</span><span class="n">status</span><span class="p">(</span><span class="s1">'/'</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'Using namenode {} for hook'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">nn</span><span class="o">.</span><span class="n">host</span><span class="p">))</span>
+ <span class="k">return</span> <span class="n">client</span>
+ <span class="k">except</span> <span class="n">HdfsError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Read operation on namenode {nn.host} failed with"</span>
+ <span class="s2">" error: {e.message}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="n">nn_hosts</span> <span class="o">=</span> <span class="p">[</span><span class="n">c</span><span class="o">.</span><span class="n">host</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">nn_connections</span><span class="p">]</span>
+ <span class="n">no_nn_error</span> <span class="o">=</span> <span class="s2">"Read operations failed on the namenodes below:</span><span class="se">\n</span><span class="s2">{}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">nn_hosts</span><span class="p">))</span>
+ <span class="k">raise</span> <span class="n">AirflowWebHDFSHookException</span><span class="p">(</span><span class="n">no_nn_error</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="WebHDFSHook.check_for_path"><a class="viewcode-back" href="../code.html#airflow.hooks.WebHDFSHook.check_for_path">[docs]</a> <span class="k">def</span> <span class="nf">check_for_path</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hdfs_path</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Check for the existence of a path in HDFS by querying FileStatus.</span>
+<span class="sd"> """</span>
+ <span class="n">c</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="k">return</span> <span class="nb">bool</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">status</span><span class="p">(</span><span class="n">hdfs_path</span><span class="p">,</span> <span class="n">strict</span><span class="o">=</span><span class="bp">False</span><span class="p">))</span></div>
+
+<div class="viewcode-block" id="WebHDFSHook.load_file"><a class="viewcode-back" href="../code.html#airflow.hooks.WebHDFSHook.load_file">[docs]</a> <span class="k">def</span> <span class="nf">load_file</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">source</span><span class="p">,</span> <span class="n">destination</span><span class="p">,</span> <span class="n">overwrite</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">parallelism</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Uploads a file to HDFS</span>
+
+<span class="sd"> :param source: Local path to file or folder. If a folder, all the files</span>
+<span class="sd"> inside of it will be uploaded (note that this implies that folders empty</span>
+<span class="sd"> of files will not be created remotely).</span>
+<span class="sd"> :type source: str</span>
+<span class="sd"> :param destination: PTarget HDFS path. If it already exists and is a</span>
+<span class="sd"> directory, files will be uploaded inside.</span>
+<span class="sd"> :type destination: str</span>
+<span class="sd"> :param overwrite: Overwrite any existing file or directory.</span>
+<span class="sd"> :type overwrite: bool</span>
+<span class="sd"> :param parallelism: Number of threads to use for parallelization. A value of</span>
+<span class="sd"> `0` (or negative) uses as many threads as there are files.</span>
+<span class="sd"> :type parallelism: int</span>
+<span class="sd"> :param \*\*kwargs: Keyword arguments forwarded to :meth:`upload`.</span>
+
+
+<span class="sd"> """</span>
+ <span class="n">c</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">c</span><span class="o">.</span><span class="n">upload</span><span class="p">(</span><span class="n">hdfs_path</span><span class="o">=</span><span class="n">destination</span><span class="p">,</span>
+ <span class="n">local_path</span><span class="o">=</span><span class="n">source</span><span class="p">,</span>
+ <span class="n">overwrite</span><span class="o">=</span><span class="n">overwrite</span><span class="p">,</span>
+ <span class="n">n_threads</span><span class="o">=</span><span class="n">parallelism</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Uploaded file {} to {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="n">destination</span><span class="p">))</span></div></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/cli.txt
----------------------------------------------------------------------
diff --git a/_sources/cli.txt b/_sources/cli.txt
new file mode 100644
index 0000000..f05cbfb
--- /dev/null
+++ b/_sources/cli.txt
@@ -0,0 +1,11 @@
+Command Line Interface
+======================
+
+Airflow has a very rich command line interface that allows for
+many types of operation on a DAG, starting services, and supporting
+development and testing.
+
+.. argparse::
+ :module: airflow.bin.cli
+ :func: get_parser
+ :prog: airflow
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/code.txt
----------------------------------------------------------------------
diff --git a/_sources/code.txt b/_sources/code.txt
new file mode 100644
index 0000000..8a48b81
--- /dev/null
+++ b/_sources/code.txt
@@ -0,0 +1,243 @@
+API Reference
+=============
+
+Operators
+---------
+Operators allow for generation of certain types of tasks that become nodes in
+the DAG when instantiated. All operators derive from BaseOperator and
+inherit many attributes and methods that way. Refer to the BaseOperator
+documentation for more details.
+
+There are 3 main types of operators:
+
+- Operators that performs an **action**, or tell another system to
+ perform an action
+- **Transfer** operators move data from one system to another
+- **Sensors** are a certain type of operator that will keep running until a
+ certain criterion is met. Examples include a specific file landing in HDFS or
+ S3, a partition appearing in Hive, or a specific time of the day. Sensors
+ are derived from ``BaseSensorOperator`` and run a poke
+ method at a specified ``poke_interval`` until it returns ``True``.
+
+BaseOperator
+''''''''''''
+All operators are derived from ``BaseOperator`` and acquire much
+functionality through inheritance. Since this is the core of the engine,
+it's worth taking the time to understand the parameters of ``BaseOperator``
+to understand the primitive features that can be leveraged in your
+DAGs.
+
+
+.. autoclass:: airflow.models.BaseOperator
+
+
+BaseSensorOperator
+'''''''''''''''''''
+All sensors are derived from ``BaseSensorOperator``. All sensors inherit
+the ``timeout`` and ``poke_interval`` on top of the ``BaseOperator``
+attributes.
+
+.. autoclass:: airflow.operators.sensors.BaseSensorOperator
+
+
+Operator API
+''''''''''''
+
+.. automodule:: airflow.operators
+ :show-inheritance:
+ :members:
+ BashOperator,
+ BranchPythonOperator,
+ TriggerDagRunOperator,
+ DummyOperator,
+ EmailOperator,
+ ExternalTaskSensor,
+ GenericTransfer,
+ HdfsSensor,
+ Hive2SambaOperator,
+ HiveOperator,
+ HivePartitionSensor,
+ HiveToDruidTransfer,
+ HiveToMySqlTransfer,
+ SimpleHttpOperator,
+ HttpSensor,
+ MetastorePartitionSensor,
+ MsSqlOperator,
+ MsSqlToHiveTransfer,
+ MySqlOperator,
+ MySqlToHiveTransfer,
+ PostgresOperator,
+ PrestoCheckOperator,
+ PrestoIntervalCheckOperator,
+ PrestoValueCheckOperator,
+ PythonOperator,
+ S3KeySensor,
+ S3ToHiveTransfer,
+ ShortCircuitOperator,
+ SlackAPIOperator,
+ SlackAPIPostOperator,
+ SqlSensor,
+ SubDagOperator,
+ TimeSensor,
+ WebHdfsSensor
+
+.. autoclass:: airflow.operators.docker_operator.DockerOperator
+
+
+Community-contributed Operators
+'''''''''''''''''''''''''''''''
+
+.. automodule:: airflow.contrib.operators
+ :show-inheritance:
+ :members:
+ BigQueryOperator,
+ BigQueryToCloudStorageOperator,
+ GoogleCloudStorageDownloadOperator,
+ SSHExecuteOperator,
+ VerticaOperator,
+ VerticaToHiveTransfer
+
+.. autoclass:: airflow.contrib.operators.QuboleOperator
+.. autoclass:: airflow.contrib.operators.hipchat_operator.HipChatAPIOperator
+.. autoclass:: airflow.contrib.operators.hipchat_operator.HipChatAPISendRoomNotificationOperator
+
+.. _macros:
+
+Macros
+---------
+Here's a list of variables and macros that can be used in templates
+
+
+Default Variables
+'''''''''''''''''
+The Airflow engine passes a few variables by default that are accessible
+in all templates
+
+================================= ====================================
+Variable Description
+================================= ====================================
+``{{ ds }}`` the execution date as ``YYYY-MM-DD``
+``{{ ds_nodash }}`` the execution date as ``YYYYMMDD``
+``{{ yesterday_ds }}`` yesterday's date as ``YYYY-MM-DD``
+``{{ yesterday_ds_nodash }}`` yesterday's date as ``YYYYMMDD``
+``{{ tomorrow_ds }}`` tomorrow's date as ``YYYY-MM-DD``
+``{{ tomorrow_ds_nodash }}`` tomorrow's date as ``YYYYMMDD``
+``{{ ts }}`` same as ``execution_date.isoformat()``
+``{{ ts_nodash }}`` same as ``ts`` without ``-`` and ``:``
+``{{ execution_date }}`` the execution_date, (datetime.datetime)
+``{{ dag }}`` the DAG object
+``{{ task }}`` the Task object
+``{{ macros }}`` a reference to the macros package, described below
+``{{ task_instance }}`` the task_instance object
+``{{ end_date }}`` same as ``{{ ds }}``
+``{{ latest_date }}`` same as ``{{ ds }}``
+``{{ ti }}`` same as ``{{ task_instance }}``
+``{{ params }}`` a reference to the user-defined params dictionary
+``{{ task_instance_key_str }}`` a unique, human-readable key to the task instance
+ formatted ``{dag_id}_{task_id}_{ds}``
+``conf`` the full configuration object located at
+ ``airflow.configuration.conf`` which
+ represents the content of your
+ ``airflow.cfg``
+``run_id`` the ``run_id`` of the current DAG run
+``dag_run`` a reference to the DagRun object
+``test_mode`` whether the task instance was called using
+ the CLI's test subcommand
+================================= ====================================
+
+Note that you can access the object's attributes and methods with simple
+dot notation. Here are some examples of what is possible:
+``{{ task.owner }}``, ``{{ task.task_id }}``, ``{{ ti.hostname }}``, ...
+Refer to the models documentation for more information on the objects'
+attributes and methods.
+
+Macros
+''''''
+Macros are a way to expose objects to your templates and live under the
+``macros`` namespace in your templates.
+
+A few commonly used libraries and methods are made available.
+
+
+================================= ====================================
+Variable Description
+================================= ====================================
+``macros.datetime`` The standard lib's ``datetime.datetime``
+``macros.timedelta`` The standard lib's ``datetime.timedelta``
+``macros.dateutil`` A reference to the ``dateutil`` package
+``macros.time`` The standard lib's ``time``
+``macros.uuid`` The standard lib's ``uuid``
+``macros.random`` The standard lib's ``random``
+================================= ====================================
+
+
+Some airflow specific macros are also defined:
+
+.. automodule:: airflow.macros
+ :show-inheritance:
+ :members:
+
+.. automodule:: airflow.macros.hive
+ :show-inheritance:
+ :members:
+
+.. _models_ref:
+
+Models
+------
+
+Models are built on top of the SQLAlchemy ORM Base class, and instances are
+persisted in the database.
+
+
+.. automodule:: airflow.models
+ :show-inheritance:
+ :members: DAG, BaseOperator, TaskInstance, DagBag, Connection
+
+Hooks
+-----
+.. automodule:: airflow.hooks
+ :show-inheritance:
+ :members:
+ DbApiHook,
+ HiveCliHook,
+ HiveMetastoreHook,
+ HiveServer2Hook,
+ HttpHook,
+ DruidHook,
+ MsSqlHook,
+ MySqlHook,
+ PostgresHook,
+ PrestoHook,
+ S3Hook,
+ SqliteHook,
+ WebHDFSHook
+
+Community contributed hooks
+'''''''''''''''''''''''''''
+
+.. automodule:: airflow.contrib.hooks
+ :show-inheritance:
+ :members:
+ BigQueryHook,
+ GoogleCloudStorageHook,
+ VerticaHook,
+ FTPHook,
+ SSHHook,
+ CloudantHook
+
+Executors
+---------
+Executors are the mechanism by which task instances get run.
+
+.. automodule:: airflow.executors
+ :show-inheritance:
+ :members: LocalExecutor, CeleryExecutor, SequentialExecutor
+
+Community-contributed executors
+'''''''''''''''''''''''''''''''
+
+.. automodule:: airflow.contrib.executors
+ :show-inheritance:
+ :members:
+ MesosExecutor
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/concepts.txt
----------------------------------------------------------------------
diff --git a/_sources/concepts.txt b/_sources/concepts.txt
new file mode 100644
index 0000000..6e15ff8
--- /dev/null
+++ b/_sources/concepts.txt
@@ -0,0 +1,758 @@
+Concepts
+########
+
+The Airflow Platform is a tool for describing, executing, and monitoring
+workflows.
+
+Core Ideas
+''''''''''
+
+DAGs
+====
+
+In Airflow, a ``DAG`` -- or a Directed Acyclic Graph -- is a collection of all
+the tasks you want to run, organized in a way that reflects their relationships
+and dependencies.
+
+For example, a simple DAG could consist of three tasks: A, B, and C. It could
+say that A has to run successfully before B can run, but C can run anytime. It
+could say that task A times out after 5 minutes, and B can be restarted up to 5
+times in case it fails. It might also say that the workflow will run every night
+at 10pm, but shouldn't start until a certain date.
+
+In this way, a DAG describes *how* you want to carry out your workflow; but
+notice that we haven't said anything about *what* we actually want to do! A, B,
+and C could be anything. Maybe A prepares data for B to analyze while C sends an
+email. Or perhaps A monitors your location so B can open your garage door while
+C turns on your house lights. The important thing is that the DAG isn't
+concerned with what its constituent tasks do; its job is to make sure that
+whatever they do happens at the right time, or in the right order, or with the
+right handling of any unexpected issues.
+
+DAGs are defined in standard Python files that are placed in Airflow's
+``DAG_FOLDER``. Airflow will execute the code in each file to dynamically build
+the ``DAG`` objects. You can have as many DAGs as you want, each describing an
+arbitrary number of tasks. In general, each one should correspond to a single
+logical workflow.
+
+Scope
+-----
+
+Airflow will load any ``DAG`` object it can import from a DAGfile. Critically,
+that means the DAG must appear in ``globals()``. Consider the following two
+DAGs. Only ``dag_1`` will be loaded; the other one only appears in a local
+scope.
+
+.. code:: python
+
+ dag_1 = DAG('this_dag_will_be_discovered')
+
+ def my_function()
+ dag_2 = DAG('but_this_dag_will_not')
+
+ my_function()
+
+Sometimes this can be put to good use. For example, a common pattern with
+``SubDagOperator`` is to define the subdag inside a function so that Airflow
+doesn't try to load it as a standalone DAG.
+
+Default Arguments
+-----------------
+
+If a dictionary of ``default_args`` is passed to a DAG, it will apply them to
+any of its operators. This makes it easy to apply a common parameter to many operators without having to type it many times.
+
+.. code:: python
+
+ default_args=dict(
+ start_date=datetime(2016, 1, 1),
+ owner='Airflow')
+
+ dag = DAG('my_dag', default_args=default_args)
+ op = DummyOperator(task_id='dummy', dag=dag)
+ print(op.owner) # Airflow
+
+Context Manager
+---------------
+
+*Added in Airflow 1.8*
+
+DAGs can be used as context managers to automatically assign new operators to that DAG.
+
+.. code:: python
+
+ with DAG('my_dag', start_date=datetime(2016, 1, 1)) as dag:
+ op = DummyOperator('op')
+
+ op.dag is dag # True
+
+Operators
+=========
+
+While DAGs describe *how* to run a workflow, ``Operators`` determine what
+actually gets done.
+
+An operator describes a single task in a workflow. Operators are usually (but
+not always) atomic, meaning they can stand on their own and don't need to share
+resources with any other operators. The DAG will make sure that operators run in
+the correct certain order; other than those dependencies, operators generally
+run independently. In fact, they may run on two completely different machines.
+
+This is a subtle but very important point: in general, if two operators need to
+share information, like a filename or small amount of data, you should consider
+combining them into a single operator. If it absolutely can't be avoided,
+Airflow does have a feature for operator cross-communication called XCom that is
+described elsewhere in this document.
+
+Airflow provides operators for many common tasks, including:
+
+- ``BashOperator`` - executes a bash command
+- ``PythonOperator`` - calls an arbitrary Python function
+- ``EmailOperator`` - sends an email
+- ``HTTPOperator`` - sends an HTTP request
+- ``SqlOperator`` - executes a SQL command
+- ``Sensor`` - waits for a certain time, file, database row, S3 key, etc...
+
+
+In addition to these basic building blocks, there are many more specific
+operators: ``DockerOperator``, ``HiveOperator``, ``S3FileTransferOperator``,
+``PrestoToMysqlOperator``, ``SlackOperator``... you get the idea!
+
+The ``airflow/contrib/`` directory contains yet more operators built by the
+community. These operators aren't always as complete or well-tested as those in
+the main distribution, but allow users to more easily add new functionality to
+the platform.
+
+Operators are only loaded by Airflow if they are assigned to a DAG.
+
+DAG Assignment
+--------------
+
+*Added in Airflow 1.8*
+
+Operators do not have to be assigned to DAGs immediately (previously ``dag`` was
+a required argument). However, once an operator is assigned to a DAG, it can not
+be transferred or unassigned. DAG assignment can be done explicitly when the
+operator is created, through deferred assignment, or even inferred from other
+operators.
+
+.. code:: python
+
+ dag = DAG('my_dag', start_date=datetime(2016, 1, 1))
+
+ # sets the DAG explicitly
+ explicit_op = DummyOperator(task_id='op1', dag=dag)
+
+ # deferred DAG assignment
+ deferred_op = DummyOperator(task_id='op2')
+ deferred_op.dag = dag
+
+ # inferred DAG assignment (linked operators must be in the same DAG)
+ inferred_op = DummyOperator(task_id='op3')
+ inferred_op.set_upstream(deferred_op)
+
+
+Bitshift Composition
+--------------------
+
+*Added in Airflow 1.8*
+
+Traditionally, operator relationships are set with the ``set_upstream()`` and
+``set_downstream()`` methods. In Airflow 1.8, this can be done with the Python
+bitshift operators ``>>`` and ``<<``. The following four statements are all
+functionally equivalent:
+
+.. code:: python
+
+ op1 >> op2
+ op1.set_downstream(op2)
+
+ op2 << op1
+ op2.set_upstream(op1)
+
+When using the bitshift to compose operators, the relationship is set in the
+direction that the bitshift operator points. For example, ``op1 >> op2`` means
+that ``op1`` runs first and ``op2`` runs second. Multiple operators can be
+composed -- keep in mind the chain is executed left-to-right and the rightmost
+object is always returned. For example:
+
+.. code:: python
+
+ op1 >> op2 >> op3 << op4
+
+is equivalent to:
+
+.. code:: python
+
+ op1.set_downstream(op2)
+ op2.set_downstream(op3)
+ op3.set_upstream(op4)
+
+For convenience, the bitshift operators can also be used with DAGs. For example:
+
+.. code:: python
+
+ dag >> op1 >> op2
+
+is equivalent to:
+
+.. code:: python
+
+ op1.dag = dag
+ op1.set_downstream(op2)
+
+We can put this all together to build a simple pipeline:
+
+.. code:: python
+
+ with DAG('my_dag', start_date=datetime(2016, 1, 1)) as dag:
+ (
+ dag
+ >> DummyOperator(task_id='dummy_1')
+ >> BashOperator(
+ task_id='bash_1',
+ bash_command='echo "HELLO!"')
+ >> PythonOperator(
+ task_id='python_1',
+ python_callable=lambda: print("GOODBYE!"))
+ )
+
+Tasks
+=====
+
+Once an operator is instantiated, it is referred to as a "task". The
+instantiation defines specific values when calling the abstract operator, and
+the parameterized task becomes a node in a DAG.
+
+Task Instances
+==============
+
+A task instance represents a specific run of a task and is characterized as the
+combination of a dag, a task, and a point in time. Task instances also have an
+indicative state, which could be "running", "success", "failed", "skipped", "up
+for retry", etc.
+
+Workflows
+=========
+
+You're now familiar with the core building blocks of Airflow.
+Some of the concepts may sound very similar, but the vocabulary can
+be conceptualized like this:
+
+- DAG: a description of the order in which work should take place
+- Operator: a class that acts as a template for carrying out some work
+- Task: a parameterized instance of an operator
+- Task Instance: a task that 1) has been assigned to a DAG and 2) has a
+ state associated with a specific run of the DAG
+
+By combining ``DAGs`` and ``Operators`` to create ``TaskInstances``, you can
+build complex workflows.
+
+Additional Functionality
+''''''''''''''''''''''''
+
+In addition to the core Airflow objects, there are a number of more complex
+features that enable behaviors like limiting simultaneous access to resources,
+cross-communication, conditional execution, and more.
+
+Hooks
+=====
+
+Hooks are interfaces to external platforms and databases like Hive, S3,
+MySQL, Postgres, HDFS, and Pig. Hooks implement a common interface when
+possible, and act as a building block for operators. They also use
+the ``airflow.models.Connection`` model to retrieve hostnames
+and authentication information. Hooks keep authentication code and
+information out of pipelines, centralized in the metadata database.
+
+Hooks are also very useful on their own to use in Python scripts,
+Airflow airflow.operators.PythonOperator, and in interactive environments
+like iPython or Jupyter Notebook.
+
+Pools
+=====
+
+Some systems can get overwhelmed when too many processes hit them at the same
+time. Airflow pools can be used to **limit the execution parallelism** on
+arbitrary sets of tasks. The list of pools is managed in the UI
+(``Menu -> Admin -> Pools``) by giving the pools a name and assigning
+it a number of worker slots. Tasks can then be associated with
+one of the existing pools by using the ``pool`` parameter when
+creating tasks (i.e., instantiating operators).
+
+.. code:: python
+
+ aggregate_db_message_job = BashOperator(
+ task_id='aggregate_db_message_job',
+ execution_timeout=timedelta(hours=3),
+ pool='ep_data_pipeline_db_msg_agg',
+ bash_command=aggregate_db_message_job_cmd,
+ dag=dag)
+ aggregate_db_message_job.set_upstream(wait_for_empty_queue)
+
+The ``pool`` parameter can
+be used in conjunction with ``priority_weight`` to define priorities
+in the queue, and which tasks get executed first as slots open up in the
+pool. The default ``priority_weight`` is ``1``, and can be bumped to any
+number. When sorting the queue to evaluate which task should be executed
+next, we use the ``priority_weight``, summed up with all of the
+``priority_weight`` values from tasks downstream from this task. You can
+use this to bump a specific important task and the whole path to that task
+gets prioritized accordingly.
+
+Tasks will be scheduled as usual while the slots fill up. Once capacity is
+reached, runnable tasks get queued and their state will show as such in the
+UI. As slots free up, queued tasks start running based on the
+``priority_weight`` (of the task and its descendants).
+
+Note that by default tasks aren't assigned to any pool and their
+execution parallelism is only limited to the executor's setting.
+
+Connections
+===========
+
+The connection information to external systems is stored in the Airflow
+metadata database and managed in the UI (``Menu -> Admin -> Connections``)
+A ``conn_id`` is defined there and hostname / login / password / schema
+information attached to it. Airflow pipelines can simply refer to the
+centrally managed ``conn_id`` without having to hard code any of this
+information anywhere.
+
+Many connections with the same ``conn_id`` can be defined and when that
+is the case, and when the **hooks** uses the ``get_connection`` method
+from ``BaseHook``, Airflow will choose one connection randomly, allowing
+for some basic load balancing and fault tolerance when used in conjunction
+with retries.
+
+Airflow also has the ability to reference connections via environment
+variables from the operating system. The environment variable needs to be
+prefixed with ``AIRFLOW_CONN_`` to be considered a connection. When
+referencing the connection in the Airflow pipeline, the ``conn_id`` should
+be the name of the variable without the prefix. For example, if the ``conn_id``
+is named ``POSTGRES_MASTER`` the environment variable should be named
+``AIRFLOW_CONN_POSTGRES_MASTER``. Airflow assumes the value returned
+from the environment variable to be in a URI format
+(e.g. ``postgres://user:password@localhost:5432/master``).
+
+Queues
+======
+
+When using the CeleryExecutor, the celery queues that tasks are sent to
+can be specified. ``queue`` is an attribute of BaseOperator, so any
+task can be assigned to any queue. The default queue for the environment
+is defined in the ``airflow.cfg``'s ``celery -> default_queue``. This defines
+the queue that tasks get assigned to when not specified, as well as which
+queue Airflow workers listen to when started.
+
+Workers can listen to one or multiple queues of tasks. When a worker is
+started (using the command ``airflow worker``), a set of comma delimited
+queue names can be specified (e.g. ``airflow worker -q spark``). This worker
+will then only pick up tasks wired to the specified queue(s).
+
+This can be useful if you need specialized workers, either from a
+resource perspective (for say very lightweight tasks where one worker
+could take thousands of tasks without a problem), or from an environment
+perspective (you want a worker running from within the Spark cluster
+itself because it needs a very specific environment and security rights).
+
+XComs
+=====
+
+XComs let tasks exchange messages, allowing more nuanced forms of control and
+shared state. The name is an abbreviation of "cross-communication". XComs are
+principally defined by a key, value, and timestamp, but also track attributes
+like the task/DAG that created the XCom and when it should become visible. Any
+object that can be pickled can be used as an XCom value, so users should make
+sure to use objects of appropriate size.
+
+XComs can be "pushed" (sent) or "pulled" (received). When a task pushes an
+XCom, it makes it generally available to other tasks. Tasks can push XComs at
+any time by calling the ``xcom_push()`` method. In addition, if a task returns
+a value (either from its Operator's ``execute()`` method, or from a
+PythonOperator's ``python_callable`` function), then an XCom containing that
+value is automatically pushed.
+
+Tasks call ``xcom_pull()`` to retrieve XComs, optionally applying filters
+based on criteria like ``key``, source ``task_ids``, and source ``dag_id``. By
+default, ``xcom_pull()`` filters for the keys that are automatically given to
+XComs when they are pushed by being returned from execute functions (as
+opposed to XComs that are pushed manually).
+
+If ``xcom_pull`` is passed a single string for ``task_ids``, then the most
+recent XCom value from that task is returned; if a list of ``task_ids`` is
+passed, then a correpsonding list of XCom values is returned.
+
+.. code:: python
+
+ # inside a PythonOperator called 'pushing_task'
+ def push_function():
+ return value
+
+ # inside another PythonOperator where provide_context=True
+ def pull_function(**context):
+ value = context['task_instance'].xcom_pull(task_ids='pushing_task')
+
+It is also possible to pull XCom directly in a template, here's an example
+of what this may look like:
+
+.. code:: sql
+
+ SELECT * FROM {{ task_instance.xcom_pull(task_ids='foo', key='table_name') }}
+
+Note that XComs are similar to `Variables`_, but are specifically designed
+for inter-task communication rather than global settings.
+
+
+Variables
+=========
+
+Variables are a generic way to store and retrieve arbitrary content or
+settings as a simple key value store within Airflow. Variables can be
+listed, created, updated and deleted from the UI (``Admin -> Variables``),
+code or CLI. While your pipeline code definition and most of your constants
+and variables should be defined in code and stored in source control,
+it can be useful to have some variables or configuration items
+accessible and modifiable through the UI.
+
+
+.. code:: python
+
+ from airflow.models import Variable
+ foo = Variable.get("foo")
+ bar = Variable.get("bar", deserialize_json=True)
+
+The second call assumes ``json`` content and will be deserialized into
+``bar``. Note that ``Variable`` is a sqlalchemy model and can be used
+as such.
+
+
+Branching
+=========
+
+Sometimes you need a workflow to branch, or only go down a certain path
+based on an arbitrary condition which is typically related to something
+that happened in an upstream task. One way to do this is by using the
+``BranchPythonOperator``.
+
+The ``BranchPythonOperator`` is much like the PythonOperator except that it
+expects a python_callable that returns a task_id. The task_id returned
+is followed, and all of the other paths are skipped.
+The task_id returned by the Python function has to be referencing a task
+directly downstream from the BranchPythonOperator task.
+
+Note that using tasks with ``depends_on_past=True`` downstream from
+``BranchPythonOperator`` is logically unsound as ``skipped`` status
+will invariably lead to block tasks that depend on their past successes.
+``skipped`` states propagates where all directly upstream tasks are
+``skipped``.
+
+If you want to skip some tasks, keep in mind that you can't have an empty
+path, if so make a dummy task.
+
+like this, the dummy task "branch_false" is skipped
+
+.. image:: img/branch_good.png
+
+Not like this, where the join task is skipped
+
+.. image:: img/branch_bad.png
+
+SubDAGs
+=======
+
+SubDAGs are perfect for repeating patterns. Defining a function that returns a
+DAG object is a nice design pattern when using Airflow.
+
+Airbnb uses the *stage-check-exchange* pattern when loading data. Data is staged
+in a temporary table, after which data quality checks are performed against
+that table. Once the checks all pass the partition is moved into the production
+table.
+
+As another example, consider the following DAG:
+
+.. image:: img/subdag_before.png
+
+We can combine all of the parallel ``task-*`` operators into a single SubDAG,
+so that the resulting DAG resembles the following:
+
+.. image:: img/subdag_after.png
+
+Note that SubDAG operators should contain a factory method that returns a DAG
+object. This will prevent the SubDAG from being treated like a separate DAG in
+the main UI. For example:
+
+.. code:: python
+
+ #dags/subdag.py
+ from airflow.models import DAG
+ from airflow.operators import DummyOperator
+
+
+ # Dag is returned by a factory method
+ def sub_dag(parent_dag_name, child_dag_name, start_date, schedule_interval):
+ dag = DAG(
+ '%s.%s' % (parent_dag_name, child_dag_name),
+ schedule_interval=schedule_interval,
+ start_date=start_date,
+ )
+
+ dummy_operator = DummyOperator(
+ task_id='dummy_task',
+ dag=dag,
+ )
+
+ return dag
+
+This SubDAG can then be referenced in your main DAG file:
+
+.. code:: python
+
+ # main_dag.py
+ from datetime import datetime, timedelta
+ from airflow.models import DAG
+ from airflow.operators import SubDagOperator
+ from dags.subdag import sub_dag
+
+
+ PARENT_DAG_NAME = 'parent_dag'
+ CHILD_DAG_NAME = 'child_dag'
+
+ main_dag = DAG(
+ dag_id=PARENT_DAG_NAME,
+ schedule_interval=timedelta(hours=1),
+ start_date=datetime(2016, 1, 1)
+ )
+
+ sub_dag = SubDagOperator(
+ subdag=sub_dag(PARENT_DAG_NAME, CHILD_DAG_NAME, main_dag.start_date,
+ main_dag.schedule_interval),
+ task_id=CHILD_DAG_NAME,
+ dag=main_dag,
+ )
+
+You can zoom into a SubDagOperator from the graph view of the main DAG to show
+the tasks contained within the SubDAG:
+
+.. image:: img/subdag_zoom.png
+
+Some other tips when using SubDAGs:
+
+- by convention, a SubDAG's ``dag_id`` should be prefixed by its parent and
+ a dot. As in ``parent.child``
+- share arguments between the main DAG and the SubDAG by passing arguments to
+ the SubDAG operator (as demonstrated above)
+- SubDAGs must have a schedule and be enabled. If the SubDAG's schedule is
+ set to ``None`` or ``@once``, the SubDAG will succeed without having done
+ anything
+- clearing a SubDagOperator also clears the state of the tasks within
+- marking success on a SubDagOperator does not affect the state of the tasks
+ within
+- refrain from using ``depends_on_past=True`` in tasks within the SubDAG as
+ this can be confusing
+- it is possible to specify an executor for the SubDAG. It is common to use
+ the SequentialExecutor if you want to run the SubDAG in-process and
+ effectively limit its parallelism to one. Using LocalExecutor can be
+ problematic as it may over-subscribe your worker, running multiple tasks in
+ a single slot
+
+See ``airflow/example_dags`` for a demonstration.
+
+SLAs
+====
+
+Service Level Agreements, or time by which a task or DAG should have
+succeeded, can be set at a task level as a ``timedelta``. If
+one or many instances have not succeeded by that time, an alert email is sent
+detailing the list of tasks that missed their SLA. The event is also recorded
+in the database and made available in the web UI under ``Browse->Missed SLAs``
+where events can be analyzed and documented.
+
+
+Trigger Rules
+=============
+
+Though the normal workflow behavior is to trigger tasks when all their
+directly upstream tasks have succeeded, Airflow allows for more complex
+dependency settings.
+
+All operators have a ``trigger_rule`` argument which defines the rule by which
+the generated task get triggered. The default value for ``trigger_rule`` is
+``all_success`` and can be defined as "trigger this task when all directly
+upstream tasks have succeeded". All other rules described here are based
+on direct parent tasks and are values that can be passed to any operator
+while creating tasks:
+
+* ``all_success``: (default) all parents have succeeded
+* ``all_failed``: all parents are in a ``failed`` or ``upstream_failed`` state
+* ``all_done``: all parents are done with their execution
+* ``one_failed``: fires as soon as at least one parent has failed, it does not wait for all parents to be done
+* ``one_success``: fires as soon as at least one parent succeeds, it does not wait for all parents to be done
+* ``dummy``: dependencies are just for show, trigger at will
+
+Note that these can be used in conjunction with ``depends_on_past`` (boolean)
+that, when set to ``True``, keeps a task from getting triggered if the
+previous schedule for the task hasn't succeeded.
+
+
+Zombies & Undeads
+=================
+
+Task instances die all the time, usually as part of their normal life cycle,
+but sometimes unexpectedly.
+
+Zombie tasks are characterized by the absence
+of an heartbeat (emitted by the job periodically) and a ``running`` status
+in the database. They can occur when a worker node can't reach the database,
+when Airflow processes are killed externally, or when a node gets rebooted
+for instance. Zombie killing is performed periodically by the scheduler's
+process.
+
+Undead processes are characterized by the existence of a process and a matching
+heartbeat, but Airflow isn't aware of this task as ``running`` in the database.
+This mismatch typically occurs as the state of the database is altered,
+most likely by deleting rows in the "Task Instances" view in the UI.
+Tasks are instructed to verify their state as part of the heartbeat routine,
+and terminate themselves upon figuring out that they are in this "undead"
+state.
+
+
+Cluster Policy
+==============
+
+Your local airflow settings file can define a ``policy`` function that
+has the ability to mutate task attributes based on other task or DAG
+attributes. It receives a single argument as a reference to task objects,
+and is expected to alter its attributes.
+
+For example, this function could apply a specific queue property when
+using a specific operator, or enforce a task timeout policy, making sure
+that no tasks run for more than 48 hours. Here's an example of what this
+may look like inside your ``airflow_settings.py``:
+
+
+.. code:: python
+
+ def policy(task):
+ if task.__class__.__name__ == 'HivePartitionSensor':
+ task.queue = "sensor_queue"
+ if task.timeout > timedelta(hours=48):
+ task.timeout = timedelta(hours=48)
+
+
+Documentation & Notes
+=====================
+
+It's possible to add documentation or notes to your dags & task objects that
+become visible in the web interface ("Graph View" for dags, "Task Details" for
+tasks). There are a set of special task attributes that get rendered as rich
+content if defined:
+
+========== ================
+attribute rendered to
+========== ================
+doc monospace
+doc_json json
+doc_yaml yaml
+doc_md markdown
+doc_rst reStructuredText
+========== ================
+
+Please note that for dags, dag_md is the only attribute interpreted.
+
+This is especially useful if your tasks are built dynamically from
+configuration files, it allows you to expose the configuration that led
+to the related tasks in Airflow.
+
+.. code:: python
+ """
+ ### My great DAG
+ """
+
+ dag = DAG('my_dag', default_args=default_args)
+ dag.doc_md = __doc__
+
+ t = BashOperator("foo", dag=dag)
+ t.doc_md = """\
+ #Title"
+ Here's a [url](www.airbnb.com)
+ """
+
+This content will get rendered as markdown respectively in the "Graph View" and
+"Task Details" pages.
+
+Jinja Templating
+================
+
+Airflow leverages the power of
+`Jinja Templating <http://jinja.pocoo.org/docs/dev/>`_ and this can be a
+powerful tool to use in combination with macros (see the :ref:`macros` section).
+
+For example, say you want to pass the execution date as an environment variable
+to a Bash script using the ``BashOperator``.
+
+.. code:: python
+
+ # The execution date as YYYY-MM-DD
+ date = "{{ ds }}"
+ t = BashOperator(
+ task_id='test_env',
+ bash_command='/tmp/test.sh ',
+ dag=dag,
+ env={'EXECUTION_DATE': date})
+
+Here, ``{{ ds }}`` is a macro, and because the ``env`` parameter of the
+``BashOperator`` is templated with Jinja, the execution date will be available
+as an environment variable named ``EXECUTION_DATE`` in your Bash script.
+
+You can use Jinja templating with every parameter that is marked as "templated"
+in the documentation.
+
+Packaged dags
+'''''''''''''
+While often you will specify dags in a single ``.py`` file it might sometimes
+be required to combine dag and its dependencies. For example, you might want
+to combine several dags together to version them together or you might want
+to manage them together or you might need an extra module that is not available
+by default on the system you are running airflow on. To allow this you can create
+a zip file that contains the dag(s) in the root of the zip file and have the extra
+modules unpacked in directories.
+
+For instance you can create a zip file that looks like this:
+
+.. code-block:: bash
+
+ my_dag1.py
+ my_dag2.py
+ package1/__init__.py
+ package1/functions.py
+
+Airflow will scan the zip file and try to load ``my_dag1.py`` and ``my_dag2.py``.
+It will not go into subdirectories as these are considered to be potential
+packages.
+
+In case you would like to add module dependencies to your DAG you basically would
+do the same, but then it is more to use a virtualenv and pip.
+
+.. code-block:: bash
+
+ virtualenv zip_dag
+ source zip_dag/bin/activate
+
+ mkdir zip_dag_contents
+ cd zip_dag_contents
+
+ pip install --install-option="--install-lib=$PWD" my_useful_package
+ cp ~/my_dag.py .
+
+ zip -r zip_dag.zip *
+
+.. note:: the zip file will be inserted at the beginning of module search list
+ (sys.path) and as such it will be available to any other code that resides
+ within the same interpreter.
+
+.. note:: packaged dags cannot be used with pickling turned on.
+
+.. note:: packaged dags cannot contain dynamic libraries (eg. libz.so) these need
+ to be available on the system if a module needs those. In other words only
+ pure python modules can be packaged.
+
[22/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/mysql_operator.html
----------------------------------------------------------------------
diff --git a/_modules/mysql_operator.html b/_modules/mysql_operator.html
new file mode 100644
index 0000000..2cb0a09
--- /dev/null
+++ b/_modules/mysql_operator.html
@@ -0,0 +1,240 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>mysql_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>mysql_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for mysql_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">MySqlHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="MySqlOperator"><a class="viewcode-back" href="../code.html#airflow.operators.MySqlOperator">[docs]</a><span class="k">class</span> <span class="nc">MySqlOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes sql code in a specific MySQL database</span>
+
+<span class="sd"> :param mysql_conn_id: reference to a specific mysql database</span>
+<span class="sd"> :type mysql_conn_id: string</span>
+<span class="sd"> :param sql: the sql code to be executed</span>
+<span class="sd"> :type sql: Can receive a str representing a sql statement,</span>
+<span class="sd"> a list of str (sql statements), or reference to a template file.</span>
+<span class="sd"> Template reference are recognized by str ending in '.sql'</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#ededed'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="n">mysql_conn_id</span><span class="o">=</span><span class="s1">'mysql_default'</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">autocommit</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">MySqlOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">mysql_conn_id</span> <span class="o">=</span> <span class="n">mysql_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">autocommit</span> <span class="o">=</span> <span class="n">autocommit</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">parameters</span> <span class="o">=</span> <span class="n">parameters</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing: '</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">))</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">MySqlHook</span><span class="p">(</span><span class="n">mysql_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_conn_id</span><span class="p">)</span>
+ <span class="n">hook</span><span class="o">.</span><span class="n">run</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">,</span>
+ <span class="n">autocommit</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">autocommit</span><span class="p">,</span>
+ <span class="n">parameters</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/mysql_to_hive.html
----------------------------------------------------------------------
diff --git a/_modules/mysql_to_hive.html b/_modules/mysql_to_hive.html
new file mode 100644
index 0000000..19ecd32
--- /dev/null
+++ b/_modules/mysql_to_hive.html
@@ -0,0 +1,316 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>mysql_to_hive — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>mysql_to_hive</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for mysql_to_hive</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">chr</span>
+<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">OrderedDict</span>
+<span class="kn">import</span> <span class="nn">unicodecsv</span> <span class="kn">as</span> <span class="nn">csv</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">from</span> <span class="nn">tempfile</span> <span class="kn">import</span> <span class="n">NamedTemporaryFile</span>
+<span class="kn">import</span> <span class="nn">MySQLdb</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">HiveCliHook</span><span class="p">,</span> <span class="n">MySqlHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="MySqlToHiveTransfer"><a class="viewcode-back" href="../code.html#airflow.operators.MySqlToHiveTransfer">[docs]</a><span class="k">class</span> <span class="nc">MySqlToHiveTransfer</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Moves data from MySql to Hive. The operator runs your query against</span>
+<span class="sd"> MySQL, stores the file locally before loading it into a Hive table.</span>
+<span class="sd"> If the ``create`` or ``recreate`` arguments are set to ``True``,</span>
+<span class="sd"> a ``CREATE TABLE`` and ``DROP TABLE`` statements are generated.</span>
+<span class="sd"> Hive data types are inferred from the cursor's metadata. Note that the</span>
+<span class="sd"> table generated in Hive uses ``STORED AS textfile``</span>
+<span class="sd"> which isn't the most efficient serialization format. If a</span>
+<span class="sd"> large amount of data is loaded and/or if the table gets</span>
+<span class="sd"> queried considerably, you may want to use this operator only to</span>
+<span class="sd"> stage the data into a temporary table before loading it into its</span>
+<span class="sd"> final destination using a ``HiveOperator``.</span>
+
+<span class="sd"> :param sql: SQL query to execute against the MySQL database</span>
+<span class="sd"> :type sql: str</span>
+<span class="sd"> :param hive_table: target Hive table, use dot notation to target a</span>
+<span class="sd"> specific database</span>
+<span class="sd"> :type hive_table: str</span>
+<span class="sd"> :param create: whether to create the table if it doesn't exist</span>
+<span class="sd"> :type create: bool</span>
+<span class="sd"> :param recreate: whether to drop and recreate the table at every</span>
+<span class="sd"> execution</span>
+<span class="sd"> :type recreate: bool</span>
+<span class="sd"> :param partition: target partition as a dict of partition columns</span>
+<span class="sd"> and values</span>
+<span class="sd"> :type partition: dict</span>
+<span class="sd"> :param delimiter: field delimiter in the file</span>
+<span class="sd"> :type delimiter: str</span>
+<span class="sd"> :param mysql_conn_id: source mysql connection</span>
+<span class="sd"> :type mysql_conn_id: str</span>
+<span class="sd"> :param hive_conn_id: destination hive connection</span>
+<span class="sd"> :type hive_conn_id: str</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,</span> <span class="s1">'partition'</span><span class="p">,</span> <span class="s1">'hive_table'</span><span class="p">)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#a0e08c'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">sql</span><span class="p">,</span>
+ <span class="n">hive_table</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="nb">chr</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span>
+ <span class="n">mysql_conn_id</span><span class="o">=</span><span class="s1">'mysql_default'</span><span class="p">,</span>
+ <span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="s1">'hive_cli_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">MySqlToHiveTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span> <span class="o">=</span> <span class="n">hive_table</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">partition</span> <span class="o">=</span> <span class="n">partition</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">create</span> <span class="o">=</span> <span class="n">create</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">recreate</span> <span class="o">=</span> <span class="n">recreate</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">delimiter</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">mysql_conn_id</span> <span class="o">=</span> <span class="n">mysql_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span> <span class="o">=</span> <span class="n">hive_cli_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">partition</span> <span class="o">=</span> <span class="n">partition</span> <span class="ow">or</span> <span class="p">{}</span>
+
+ <span class="nd">@classmethod</span>
+ <span class="k">def</span> <span class="nf">type_map</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">mysql_type</span><span class="p">):</span>
+ <span class="n">t</span> <span class="o">=</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">constants</span><span class="o">.</span><span class="n">FIELD_TYPE</span>
+ <span class="n">d</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="n">t</span><span class="o">.</span><span class="n">BIT</span><span class="p">:</span> <span class="s1">'INT'</span><span class="p">,</span>
+ <span class="n">t</span><span class="o">.</span><span class="n">DECIMAL</span><span class="p">:</span> <span class="s1">'DOUBLE'</span><span class="p">,</span>
+ <span class="n">t</span><span class="o">.</span><span class="n">DOUBLE</span><span class="p">:</span> <span class="s1">'DOUBLE'</span><span class="p">,</span>
+ <span class="n">t</span><span class="o">.</span><span class="n">FLOAT</span><span class="p">:</span> <span class="s1">'DOUBLE'</span><span class="p">,</span>
+ <span class="n">t</span><span class="o">.</span><span class="n">INT24</span><span class="p">:</span> <span class="s1">'INT'</span><span class="p">,</span>
+ <span class="n">t</span><span class="o">.</span><span class="n">LONG</span><span class="p">:</span> <span class="s1">'INT'</span><span class="p">,</span>
+ <span class="n">t</span><span class="o">.</span><span class="n">LONGLONG</span><span class="p">:</span> <span class="s1">'BIGINT'</span><span class="p">,</span>
+ <span class="n">t</span><span class="o">.</span><span class="n">SHORT</span><span class="p">:</span> <span class="s1">'INT'</span><span class="p">,</span>
+ <span class="n">t</span><span class="o">.</span><span class="n">YEAR</span><span class="p">:</span> <span class="s1">'INT'</span><span class="p">,</span>
+ <span class="p">}</span>
+ <span class="k">return</span> <span class="n">d</span><span class="p">[</span><span class="n">mysql_type</span><span class="p">]</span> <span class="k">if</span> <span class="n">mysql_type</span> <span class="ow">in</span> <span class="n">d</span> <span class="k">else</span> <span class="s1">'STRING'</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">hive</span> <span class="o">=</span> <span class="n">HiveCliHook</span><span class="p">(</span><span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span><span class="p">)</span>
+ <span class="n">mysql</span> <span class="o">=</span> <span class="n">MySqlHook</span><span class="p">(</span><span class="n">mysql_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_conn_id</span><span class="p">)</span>
+
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Dumping MySQL query results to local file"</span><span class="p">)</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="n">mysql</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">cursor</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
+ <span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+ <span class="k">with</span> <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="s2">"wb"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+ <span class="n">csv_writer</span> <span class="o">=</span> <span class="n">csv</span><span class="o">.</span><span class="n">writer</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">"utf-8"</span><span class="p">)</span>
+ <span class="n">field_dict</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">()</span>
+ <span class="k">for</span> <span class="n">field</span> <span class="ow">in</span> <span class="n">cursor</span><span class="o">.</span><span class="n">description</span><span class="p">:</span>
+ <span class="n">field_dict</span><span class="p">[</span><span class="n">field</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">type_map</span><span class="p">(</span><span class="n">field</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
+ <span class="n">csv_writer</span><span class="o">.</span><span class="n">writerows</span><span class="p">(</span><span class="n">cursor</span><span class="p">)</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
+ <span class="n">cursor</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Loading file into Hive"</span><span class="p">)</span>
+ <span class="n">hive</span><span class="o">.</span><span class="n">load_file</span><span class="p">(</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span><span class="p">,</span>
+ <span class="n">field_dict</span><span class="o">=</span><span class="n">field_dict</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">create</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">partition</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">recreate</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/postgres_hook.html
----------------------------------------------------------------------
diff --git a/_modules/postgres_hook.html b/_modules/postgres_hook.html
new file mode 100644
index 0000000..e0aa9eb
--- /dev/null
+++ b/_modules/postgres_hook.html
@@ -0,0 +1,236 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>postgres_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>postgres_hook</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for postgres_hook</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">psycopg2</span>
+<span class="kn">import</span> <span class="nn">psycopg2.extensions</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks.dbapi_hook</span> <span class="kn">import</span> <span class="n">DbApiHook</span>
+
+
+<div class="viewcode-block" id="PostgresHook"><a class="viewcode-back" href="../code.html#airflow.hooks.PostgresHook">[docs]</a><span class="k">class</span> <span class="nc">PostgresHook</span><span class="p">(</span><span class="n">DbApiHook</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> Interact with Postgres.</span>
+<span class="sd"> You can specify ssl parameters in the extra field of your connection</span>
+<span class="sd"> as ``{"sslmode": "require", "sslcert": "/path/to/cert.pem", etc}``.</span>
+<span class="sd"> '''</span>
+ <span class="n">conn_name_attr</span> <span class="o">=</span> <span class="s1">'postgres_conn_id'</span>
+ <span class="n">default_conn_name</span> <span class="o">=</span> <span class="s1">'postgres_default'</span>
+ <span class="n">supports_autocommit</span> <span class="o">=</span> <span class="bp">False</span>
+
+ <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">postgres_conn_id</span><span class="p">)</span>
+ <span class="n">conn_args</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span>
+ <span class="n">host</span><span class="o">=</span><span class="n">conn</span><span class="o">.</span><span class="n">host</span><span class="p">,</span>
+ <span class="n">user</span><span class="o">=</span><span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">,</span>
+ <span class="n">password</span><span class="o">=</span><span class="n">conn</span><span class="o">.</span><span class="n">password</span><span class="p">,</span>
+ <span class="n">dbname</span><span class="o">=</span><span class="n">conn</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span>
+ <span class="n">port</span><span class="o">=</span><span class="n">conn</span><span class="o">.</span><span class="n">port</span><span class="p">)</span>
+ <span class="c1"># check for ssl parameters in conn.extra</span>
+ <span class="k">for</span> <span class="n">arg_name</span><span class="p">,</span> <span class="n">arg_val</span> <span class="ow">in</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
+ <span class="k">if</span> <span class="n">arg_name</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">'sslmode'</span><span class="p">,</span> <span class="s1">'sslcert'</span><span class="p">,</span> <span class="s1">'sslkey'</span><span class="p">,</span> <span class="s1">'sslrootcert'</span><span class="p">,</span> <span class="s1">'sslcrl'</span><span class="p">]:</span>
+ <span class="n">conn_args</span><span class="p">[</span><span class="n">arg_name</span><span class="p">]</span> <span class="o">=</span> <span class="n">arg_val</span>
+ <span class="n">psycopg2_conn</span> <span class="o">=</span> <span class="n">psycopg2</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="o">**</span><span class="n">conn_args</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">psycopg2_conn</span><span class="o">.</span><span class="n">server_version</span> <span class="o"><</span> <span class="mi">70400</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">supports_autocommit</span> <span class="o">=</span> <span class="bp">True</span>
+ <span class="k">return</span> <span class="n">psycopg2_conn</span>
+
+ <span class="nd">@staticmethod</span>
+ <span class="k">def</span> <span class="nf">_serialize_cell</span><span class="p">(</span><span class="n">cell</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">psycopg2</span><span class="o">.</span><span class="n">extensions</span><span class="o">.</span><span class="n">adapt</span><span class="p">(</span><span class="n">cell</span><span class="p">)</span><span class="o">.</span><span class="n">getquoted</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/postgres_operator.html
----------------------------------------------------------------------
diff --git a/_modules/postgres_operator.html b/_modules/postgres_operator.html
new file mode 100644
index 0000000..a9ec233
--- /dev/null
+++ b/_modules/postgres_operator.html
@@ -0,0 +1,239 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>postgres_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>postgres_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for postgres_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">PostgresHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="PostgresOperator"><a class="viewcode-back" href="../code.html#airflow.operators.PostgresOperator">[docs]</a><span class="k">class</span> <span class="nc">PostgresOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes sql code in a specific Postgres database</span>
+
+<span class="sd"> :param postgres_conn_id: reference to a specific postgres database</span>
+<span class="sd"> :type postgres_conn_id: string</span>
+<span class="sd"> :param sql: the sql code to be executed</span>
+<span class="sd"> :type sql: Can receive a str representing a sql statement,</span>
+<span class="sd"> a list of str (sql statements), or reference to a template file.</span>
+<span class="sd"> Template reference are recognized by str ending in '.sql'</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#ededed'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span>
+ <span class="n">postgres_conn_id</span><span class="o">=</span><span class="s1">'postgres_default'</span><span class="p">,</span> <span class="n">autocommit</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">parameters</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">PostgresOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">postgres_conn_id</span> <span class="o">=</span> <span class="n">postgres_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">autocommit</span> <span class="o">=</span> <span class="n">autocommit</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">parameters</span> <span class="o">=</span> <span class="n">parameters</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing: '</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">PostgresHook</span><span class="p">(</span><span class="n">postgres_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">postgres_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">autocommit</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/presto_check_operator.html
----------------------------------------------------------------------
diff --git a/_modules/presto_check_operator.html b/_modules/presto_check_operator.html
new file mode 100644
index 0000000..c99d551
--- /dev/null
+++ b/_modules/presto_check_operator.html
@@ -0,0 +1,303 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>presto_check_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>presto_check_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for presto_check_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">PrestoHook</span>
+<span class="kn">from</span> <span class="nn">airflow.operators</span> <span class="kn">import</span> <span class="n">CheckOperator</span><span class="p">,</span> <span class="n">ValueCheckOperator</span><span class="p">,</span> <span class="n">IntervalCheckOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="PrestoCheckOperator"><a class="viewcode-back" href="../code.html#airflow.operators.PrestoCheckOperator">[docs]</a><span class="k">class</span> <span class="nc">PrestoCheckOperator</span><span class="p">(</span><span class="n">CheckOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Performs checks against Presto. The ``PrestoCheckOperator`` expects</span>
+<span class="sd"> a sql query that will return a single row. Each value on that</span>
+<span class="sd"> first row is evaluated using python ``bool`` casting. If any of the</span>
+<span class="sd"> values return ``False`` the check is failed and errors out.</span>
+
+<span class="sd"> Note that Python bool casting evals the following as ``False``:</span>
+<span class="sd"> * False</span>
+<span class="sd"> * 0</span>
+<span class="sd"> * Empty string (``""``)</span>
+<span class="sd"> * Empty list (``[]``)</span>
+<span class="sd"> * Empty dictionary or set (``{}``)</span>
+
+<span class="sd"> Given a query like ``SELECT COUNT(*) FROM foo``, it will fail only if</span>
+<span class="sd"> the count ``== 0``. You can craft much more complex query that could,</span>
+<span class="sd"> for instance, check that the table has the same number of rows as</span>
+<span class="sd"> the source table upstream, or that the count of today's partition is</span>
+<span class="sd"> greater than yesterday's partition, or that a set of metrics are less</span>
+<span class="sd"> than 3 standard deviation for the 7 day average.</span>
+
+<span class="sd"> This operator can be used as a data quality check in your pipeline, and</span>
+<span class="sd"> depending on where you put it in your DAG, you have the choice to</span>
+<span class="sd"> stop the critical path, preventing from</span>
+<span class="sd"> publishing dubious data, or on the side and receive email alterts</span>
+<span class="sd"> without stopping the progress of the DAG.</span>
+
+<span class="sd"> :param sql: the sql to be executed</span>
+<span class="sd"> :type sql: string</span>
+<span class="sd"> :param presto_conn_id: reference to the Presto database</span>
+<span class="sd"> :type presto_conn_id: string</span>
+<span class="sd"> """</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span>
+ <span class="n">presto_conn_id</span><span class="o">=</span><span class="s1">'presto_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">PrestoCheckOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">sql</span><span class="o">=</span><span class="n">sql</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">presto_conn_id</span> <span class="o">=</span> <span class="n">presto_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
+
+ <span class="k">def</span> <span class="nf">get_db_hook</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">PrestoHook</span><span class="p">(</span><span class="n">presto_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">presto_conn_id</span><span class="p">)</span></div>
+
+
+<div class="viewcode-block" id="PrestoValueCheckOperator"><a class="viewcode-back" href="../code.html#airflow.operators.PrestoValueCheckOperator">[docs]</a><span class="k">class</span> <span class="nc">PrestoValueCheckOperator</span><span class="p">(</span><span class="n">ValueCheckOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Performs a simple value check using sql code.</span>
+
+<span class="sd"> :param sql: the sql to be executed</span>
+<span class="sd"> :type sql: string</span>
+<span class="sd"> :param presto_conn_id: reference to the Presto database</span>
+<span class="sd"> :type presto_conn_id: string</span>
+<span class="sd"> """</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="n">pass_value</span><span class="p">,</span> <span class="n">tolerance</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">presto_conn_id</span><span class="o">=</span><span class="s1">'presto_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">PrestoValueCheckOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="n">pass_value</span><span class="p">,</span> <span class="n">tolerance</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">presto_conn_id</span> <span class="o">=</span> <span class="n">presto_conn_id</span>
+
+ <span class="k">def</span> <span class="nf">get_db_hook</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">PrestoHook</span><span class="p">(</span><span class="n">presto_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">presto_conn_id</span><span class="p">)</span></div>
+
+
+<div class="viewcode-block" id="PrestoIntervalCheckOperator"><a class="viewcode-back" href="../code.html#airflow.operators.PrestoIntervalCheckOperator">[docs]</a><span class="k">class</span> <span class="nc">PrestoIntervalCheckOperator</span><span class="p">(</span><span class="n">IntervalCheckOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Checks that the values of metrics given as SQL expressions are within</span>
+<span class="sd"> a certain tolerance of the ones from days_back before.</span>
+
+<span class="sd"> :param table: the table name</span>
+<span class="sd"> :type table: str</span>
+<span class="sd"> :param days_back: number of days between ds and the ds we want to check</span>
+<span class="sd"> against. Defaults to 7 days</span>
+<span class="sd"> :type days_back: int</span>
+<span class="sd"> :param metrics_threshold: a dictionary of ratios indexed by metrics</span>
+<span class="sd"> :type metrics_threshold: dict</span>
+<span class="sd"> :param presto_conn_id: reference to the Presto database</span>
+<span class="sd"> :type presto_conn_id: string</span>
+<span class="sd"> """</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">metrics_thresholds</span><span class="p">,</span>
+ <span class="n">date_filter_column</span><span class="o">=</span><span class="s1">'ds'</span><span class="p">,</span> <span class="n">days_back</span><span class="o">=-</span><span class="mi">7</span><span class="p">,</span>
+ <span class="n">presto_conn_id</span><span class="o">=</span><span class="s1">'presto_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">PrestoIntervalCheckOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span>
+ <span class="n">table</span><span class="p">,</span> <span class="n">metrics_thresholds</span><span class="p">,</span> <span class="n">date_filter_column</span><span class="p">,</span> <span class="n">days_back</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">presto_conn_id</span> <span class="o">=</span> <span class="n">presto_conn_id</span>
+
+ <span class="k">def</span> <span class="nf">get_db_hook</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">PrestoHook</span><span class="p">(</span><span class="n">presto_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">presto_conn_id</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[09/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/js/modernizr.min.js
----------------------------------------------------------------------
diff --git a/_static/js/modernizr.min.js b/_static/js/modernizr.min.js
new file mode 100644
index 0000000..f65d479
--- /dev/null
+++ b/_static/js/modernizr.min.js
@@ -0,0 +1,4 @@
+/* Modernizr 2.6.2 (Custom Build) | MIT & BSD
+ * Build: http://modernizr.com/download/#-fontface-backgroundsize-borderimage-borderradius-boxshadow-flexbox-hsla-multiplebgs-opacity-rgba-textshadow-cssanimations-csscolumns-generatedcontent-cssgradients-cssreflections-csstransforms-csstransforms3d-csstransitions-applicationcache-canvas-canvastext-draganddrop-hashchange-history-audio-video-indexeddb-input-inputtypes-localstorage-postmessage-sessionstorage-websockets-websqldatabase-webworkers-geolocation-inlinesvg-smil-svg-svgclippaths-touch-webgl-shiv-mq-cssclasses-addtest-prefixed-teststyles-testprop-testallprops-hasevent-prefixes-domprefixes-load
+ */
+;window.Modernizr=function(a,b,c){function D(a){j.cssText=a}function E(a,b){return D(n.join(a+";")+(b||""))}function F(a,b){return typeof a===b}function G(a,b){return!!~(""+a).indexOf(b)}function H(a,b){for(var d in a){var e=a[d];if(!G(e,"-")&&j[e]!==c)return b=="pfx"?e:!0}return!1}function I(a,b,d){for(var e in a){var f=b[a[e]];if(f!==c)return d===!1?a[e]:F(f,"function")?f.bind(d||b):f}return!1}function J(a,b,c){var d=a.charAt(0).toUpperCase()+a.slice(1),e=(a+" "+p.join(d+" ")+d).split(" ");return F(b,"string")||F(b,"undefined")?H(e,b):(e=(a+" "+q.join(d+" ")+d).split(" "),I(e,b,c))}function K(){e.input=function(c){for(var d=0,e=c.length;d<e;d++)u[c[d]]=c[d]in k;return u.list&&(u.list=!!b.createElement("datalist")&&!!a.HTMLDataListElement),u}("autocomplete autofocus list placeholder max min multiple pattern required step".split(" ")),e.inputtypes=function(a){for(var d=0,e,f,h,i=a.length;d<i;d++)k.setAttribute("type",f=a[d]),e=k.type!=="text",e&&(k.value=l,k.style.cssText="position:
absolute;visibility:hidden;",/^range$/.test(f)&&k.style.WebkitAppearance!==c?(g.appendChild(k),h=b.defaultView,e=h.getComputedStyle&&h.getComputedStyle(k,null).WebkitAppearance!=="textfield"&&k.offsetHeight!==0,g.removeChild(k)):/^(search|tel)$/.test(f)||(/^(url|email)$/.test(f)?e=k.checkValidity&&k.checkValidity()===!1:e=k.value!=l)),t[a[d]]=!!e;return t}("search tel url email datetime date month week time datetime-local number range color".split(" "))}var d="2.6.2",e={},f=!0,g=b.documentElement,h="modernizr",i=b.createElement(h),j=i.style,k=b.createElement("input"),l=":)",m={}.toString,n=" -webkit- -moz- -o- -ms- ".split(" "),o="Webkit Moz O ms",p=o.split(" "),q=o.toLowerCase().split(" "),r={svg:"http://www.w3.org/2000/svg"},s={},t={},u={},v=[],w=v.slice,x,y=function(a,c,d,e){var f,i,j,k,l=b.createElement("div"),m=b.body,n=m||b.createElement("body");if(parseInt(d,10))while(d--)j=b.createElement("div"),j.id=e?e[d]:h+(d+1),l.appendChild(j);return f=["­",'<style id="s',h,'">',a,
"</style>"].join(""),l.id=h,(m?l:n).innerHTML+=f,n.appendChild(l),m||(n.style.background="",n.style.overflow="hidden",k=g.style.overflow,g.style.overflow="hidden",g.appendChild(n)),i=c(l,a),m?l.parentNode.removeChild(l):(n.parentNode.removeChild(n),g.style.overflow=k),!!i},z=function(b){var c=a.matchMedia||a.msMatchMedia;if(c)return c(b).matches;var d;return y("@media "+b+" { #"+h+" { position: absolute; } }",function(b){d=(a.getComputedStyle?getComputedStyle(b,null):b.currentStyle)["position"]=="absolute"}),d},A=function(){function d(d,e){e=e||b.createElement(a[d]||"div"),d="on"+d;var f=d in e;return f||(e.setAttribute||(e=b.createElement("div")),e.setAttribute&&e.removeAttribute&&(e.setAttribute(d,""),f=F(e[d],"function"),F(e[d],"undefined")||(e[d]=c),e.removeAttribute(d))),e=null,f}var a={select:"input",change:"input",submit:"form",reset:"form",error:"img",load:"img",abort:"img"};return d}(),B={}.hasOwnProperty,C;!F(B,"undefined")&&!F(B.call,"undefined")?C=function(a,b){return B.
call(a,b)}:C=function(a,b){return b in a&&F(a.constructor.prototype[b],"undefined")},Function.prototype.bind||(Function.prototype.bind=function(b){var c=this;if(typeof c!="function")throw new TypeError;var d=w.call(arguments,1),e=function(){if(this instanceof e){var a=function(){};a.prototype=c.prototype;var f=new a,g=c.apply(f,d.concat(w.call(arguments)));return Object(g)===g?g:f}return c.apply(b,d.concat(w.call(arguments)))};return e}),s.flexbox=function(){return J("flexWrap")},s.canvas=function(){var a=b.createElement("canvas");return!!a.getContext&&!!a.getContext("2d")},s.canvastext=function(){return!!e.canvas&&!!F(b.createElement("canvas").getContext("2d").fillText,"function")},s.webgl=function(){return!!a.WebGLRenderingContext},s.touch=function(){var c;return"ontouchstart"in a||a.DocumentTouch&&b instanceof DocumentTouch?c=!0:y(["@media (",n.join("touch-enabled),("),h,")","{#modernizr{top:9px;position:absolute}}"].join(""),function(a){c=a.offsetTop===9}),c},s.geolocation=funct
ion(){return"geolocation"in navigator},s.postmessage=function(){return!!a.postMessage},s.websqldatabase=function(){return!!a.openDatabase},s.indexedDB=function(){return!!J("indexedDB",a)},s.hashchange=function(){return A("hashchange",a)&&(b.documentMode===c||b.documentMode>7)},s.history=function(){return!!a.history&&!!history.pushState},s.draganddrop=function(){var a=b.createElement("div");return"draggable"in a||"ondragstart"in a&&"ondrop"in a},s.websockets=function(){return"WebSocket"in a||"MozWebSocket"in a},s.rgba=function(){return D("background-color:rgba(150,255,150,.5)"),G(j.backgroundColor,"rgba")},s.hsla=function(){return D("background-color:hsla(120,40%,100%,.5)"),G(j.backgroundColor,"rgba")||G(j.backgroundColor,"hsla")},s.multiplebgs=function(){return D("background:url(https://),url(https://),red url(https://)"),/(url\s*\(.*?){3}/.test(j.background)},s.backgroundsize=function(){return J("backgroundSize")},s.borderimage=function(){return J("borderImage")},s.borderradius=fun
ction(){return J("borderRadius")},s.boxshadow=function(){return J("boxShadow")},s.textshadow=function(){return b.createElement("div").style.textShadow===""},s.opacity=function(){return E("opacity:.55"),/^0.55$/.test(j.opacity)},s.cssanimations=function(){return J("animationName")},s.csscolumns=function(){return J("columnCount")},s.cssgradients=function(){var a="background-image:",b="gradient(linear,left top,right bottom,from(#9f9),to(white));",c="linear-gradient(left top,#9f9, white);";return D((a+"-webkit- ".split(" ").join(b+a)+n.join(c+a)).slice(0,-a.length)),G(j.backgroundImage,"gradient")},s.cssreflections=function(){return J("boxReflect")},s.csstransforms=function(){return!!J("transform")},s.csstransforms3d=function(){var a=!!J("perspective");return a&&"webkitPerspective"in g.style&&y("@media (transform-3d),(-webkit-transform-3d){#modernizr{left:9px;position:absolute;height:3px;}}",function(b,c){a=b.offsetLeft===9&&b.offsetHeight===3}),a},s.csstransitions=function(){return J("
transition")},s.fontface=function(){var a;return y('@font-face {font-family:"font";src:url("https://")}',function(c,d){var e=b.getElementById("smodernizr"),f=e.sheet||e.styleSheet,g=f?f.cssRules&&f.cssRules[0]?f.cssRules[0].cssText:f.cssText||"":"";a=/src/i.test(g)&&g.indexOf(d.split(" ")[0])===0}),a},s.generatedcontent=function(){var a;return y(["#",h,"{font:0/0 a}#",h,':after{content:"',l,'";visibility:hidden;font:3px/1 a}'].join(""),function(b){a=b.offsetHeight>=3}),a},s.video=function(){var a=b.createElement("video"),c=!1;try{if(c=!!a.canPlayType)c=new Boolean(c),c.ogg=a.canPlayType('video/ogg; codecs="theora"').replace(/^no$/,""),c.h264=a.canPlayType('video/mp4; codecs="avc1.42E01E"').replace(/^no$/,""),c.webm=a.canPlayType('video/webm; codecs="vp8, vorbis"').replace(/^no$/,"")}catch(d){}return c},s.audio=function(){var a=b.createElement("audio"),c=!1;try{if(c=!!a.canPlayType)c=new Boolean(c),c.ogg=a.canPlayType('audio/ogg; codecs="vorbis"').replace(/^no$/,""),c.mp3=a.canPlayTy
pe("audio/mpeg;").replace(/^no$/,""),c.wav=a.canPlayType('audio/wav; codecs="1"').replace(/^no$/,""),c.m4a=(a.canPlayType("audio/x-m4a;")||a.canPlayType("audio/aac;")).replace(/^no$/,"")}catch(d){}return c},s.localstorage=function(){try{return localStorage.setItem(h,h),localStorage.removeItem(h),!0}catch(a){return!1}},s.sessionstorage=function(){try{return sessionStorage.setItem(h,h),sessionStorage.removeItem(h),!0}catch(a){return!1}},s.webworkers=function(){return!!a.Worker},s.applicationcache=function(){return!!a.applicationCache},s.svg=function(){return!!b.createElementNS&&!!b.createElementNS(r.svg,"svg").createSVGRect},s.inlinesvg=function(){var a=b.createElement("div");return a.innerHTML="<svg/>",(a.firstChild&&a.firstChild.namespaceURI)==r.svg},s.smil=function(){return!!b.createElementNS&&/SVGAnimate/.test(m.call(b.createElementNS(r.svg,"animate")))},s.svgclippaths=function(){return!!b.createElementNS&&/SVGClipPath/.test(m.call(b.createElementNS(r.svg,"clipPath")))};for(var L
in s)C(s,L)&&(x=L.toLowerCase(),e[x]=s[L](),v.push((e[x]?"":"no-")+x));return e.input||K(),e.addTest=function(a,b){if(typeof a=="object")for(var d in a)C(a,d)&&e.addTest(d,a[d]);else{a=a.toLowerCase();if(e[a]!==c)return e;b=typeof b=="function"?b():b,typeof f!="undefined"&&f&&(g.className+=" "+(b?"":"no-")+a),e[a]=b}return e},D(""),i=k=null,function(a,b){function k(a,b){var c=a.createElement("p"),d=a.getElementsByTagName("head")[0]||a.documentElement;return c.innerHTML="x<style>"+b+"</style>",d.insertBefore(c.lastChild,d.firstChild)}function l(){var a=r.elements;return typeof a=="string"?a.split(" "):a}function m(a){var b=i[a[g]];return b||(b={},h++,a[g]=h,i[h]=b),b}function n(a,c,f){c||(c=b);if(j)return c.createElement(a);f||(f=m(c));var g;return f.cache[a]?g=f.cache[a].cloneNode():e.test(a)?g=(f.cache[a]=f.createElem(a)).cloneNode():g=f.createElem(a),g.canHaveChildren&&!d.test(a)?f.frag.appendChild(g):g}function o(a,c){a||(a=b);if(j)return a.createDocumentFragment();c=c||m(a);var
d=c.frag.cloneNode(),e=0,f=l(),g=f.length;for(;e<g;e++)d.createElement(f[e]);return d}function p(a,b){b.cache||(b.cache={},b.createElem=a.createElement,b.createFrag=a.createDocumentFragment,b.frag=b.createFrag()),a.createElement=function(c){return r.shivMethods?n(c,a,b):b.createElem(c)},a.createDocumentFragment=Function("h,f","return function(){var n=f.cloneNode(),c=n.createElement;h.shivMethods&&("+l().join().replace(/\w+/g,function(a){return b.createElem(a),b.frag.createElement(a),'c("'+a+'")'})+");return n}")(r,b.frag)}function q(a){a||(a=b);var c=m(a);return r.shivCSS&&!f&&!c.hasCSS&&(c.hasCSS=!!k(a,"article,aside,figcaption,figure,footer,header,hgroup,nav,section{display:block}mark{background:#FF0;color:#000}")),j||p(a,c),a}var c=a.html5||{},d=/^<|^(?:button|map|select|textarea|object|iframe|option|optgroup)$/i,e=/^(?:a|b|code|div|fieldset|h1|h2|h3|h4|h5|h6|i|label|li|ol|p|q|span|strong|style|table|tbody|td|th|tr|ul)$/i,f,g="_html5shiv",h=0,i={},j;(function(){try{var a=b.create
Element("a");a.innerHTML="<xyz></xyz>",f="hidden"in a,j=a.childNodes.length==1||function(){b.createElement("a");var a=b.createDocumentFragment();return typeof a.cloneNode=="undefined"||typeof a.createDocumentFragment=="undefined"||typeof a.createElement=="undefined"}()}catch(c){f=!0,j=!0}})();var r={elements:c.elements||"abbr article aside audio bdi canvas data datalist details figcaption figure footer header hgroup mark meter nav output progress section summary time video",shivCSS:c.shivCSS!==!1,supportsUnknownElements:j,shivMethods:c.shivMethods!==!1,type:"default",shivDocument:q,createElement:n,createDocumentFragment:o};a.html5=r,q(b)}(this,b),e._version=d,e._prefixes=n,e._domPrefixes=q,e._cssomPrefixes=p,e.mq=z,e.hasEvent=A,e.testProp=function(a){return H([a])},e.testAllProps=J,e.testStyles=y,e.prefixed=function(a,b,c){return b?J(a,b,c):J(a,"pfx")},g.className=g.className.replace(/(^|\s)no-js(\s|$)/,"$1$2")+(f?" js "+v.join(" "):""),e}(this,this.document),function(a,b,c){functio
n d(a){return"[object Function]"==o.call(a)}function e(a){return"string"==typeof a}function f(){}function g(a){return!a||"loaded"==a||"complete"==a||"uninitialized"==a}function h(){var a=p.shift();q=1,a?a.t?m(function(){("c"==a.t?B.injectCss:B.injectJs)(a.s,0,a.a,a.x,a.e,1)},0):(a(),h()):q=0}function i(a,c,d,e,f,i,j){function k(b){if(!o&&g(l.readyState)&&(u.r=o=1,!q&&h(),l.onload=l.onreadystatechange=null,b)){"img"!=a&&m(function(){t.removeChild(l)},50);for(var d in y[c])y[c].hasOwnProperty(d)&&y[c][d].onload()}}var j=j||B.errorTimeout,l=b.createElement(a),o=0,r=0,u={t:d,s:c,e:f,a:i,x:j};1===y[c]&&(r=1,y[c]=[]),"object"==a?l.data=c:(l.src=c,l.type=a),l.width=l.height="0",l.onerror=l.onload=l.onreadystatechange=function(){k.call(this,r)},p.splice(e,0,u),"img"!=a&&(r||2===y[c]?(t.insertBefore(l,s?null:n),m(k,j)):y[c].push(l))}function j(a,b,c,d,f){return q=0,b=b||"j",e(a)?i("c"==b?v:u,a,b,this.i++,c,d,f):(p.splice(this.i++,0,a),1==p.length&&h()),this}function k(){var a=B;return a.load
er={load:j,i:0},a}var l=b.documentElement,m=a.setTimeout,n=b.getElementsByTagName("script")[0],o={}.toString,p=[],q=0,r="MozAppearance"in l.style,s=r&&!!b.createRange().compareNode,t=s?l:n.parentNode,l=a.opera&&"[object Opera]"==o.call(a.opera),l=!!b.attachEvent&&!l,u=r?"object":l?"script":"img",v=l?"script":u,w=Array.isArray||function(a){return"[object Array]"==o.call(a)},x=[],y={},z={timeout:function(a,b){return b.length&&(a.timeout=b[0]),a}},A,B;B=function(a){function b(a){var a=a.split("!"),b=x.length,c=a.pop(),d=a.length,c={url:c,origUrl:c,prefixes:a},e,f,g;for(f=0;f<d;f++)g=a[f].split("="),(e=z[g.shift()])&&(c=e(c,g));for(f=0;f<b;f++)c=x[f](c);return c}function g(a,e,f,g,h){var i=b(a),j=i.autoCallback;i.url.split(".").pop().split("?").shift(),i.bypass||(e&&(e=d(e)?e:e[a]||e[g]||e[a.split("/").pop().split("?")[0]]),i.instead?i.instead(a,e,f,g,h):(y[i.url]?i.noexec=!0:y[i.url]=1,f.load(i.url,i.forceCSS||!i.forceJS&&"css"==i.url.split(".").pop().split("?").shift()?"c":c,i.noexec,
i.attrs,i.timeout),(d(e)||d(j))&&f.load(function(){k(),e&&e(i.origUrl,h,g),j&&j(i.origUrl,h,g),y[i.url]=2})))}function h(a,b){function c(a,c){if(a){if(e(a))c||(j=function(){var a=[].slice.call(arguments);k.apply(this,a),l()}),g(a,j,b,0,h);else if(Object(a)===a)for(n in m=function(){var b=0,c;for(c in a)a.hasOwnProperty(c)&&b++;return b}(),a)a.hasOwnProperty(n)&&(!c&&!--m&&(d(j)?j=function(){var a=[].slice.call(arguments);k.apply(this,a),l()}:j[n]=function(a){return function(){var b=[].slice.call(arguments);a&&a.apply(this,b),l()}}(k[n])),g(a[n],j,b,n,h))}else!c&&l()}var h=!!a.test,i=a.load||a.both,j=a.callback||f,k=j,l=a.complete||f,m,n;c(h?a.yep:a.nope,!!i),i&&c(i)}var i,j,l=this.yepnope.loader;if(e(a))g(a,0,l,0);else if(w(a))for(i=0;i<a.length;i++)j=a[i],e(j)?g(j,0,l,0):w(j)?B(j):Object(j)===j&&h(j,l);else Object(a)===a&&h(a,l)},B.addPrefix=function(a,b){z[a]=b},B.addFilter=function(a){x.push(a)},B.errorTimeout=1e4,null==b.readyState&&b.addEventListener&&(b.readyState="loading",b.
addEventListener("DOMContentLoaded",A=function(){b.removeEventListener("DOMContentLoaded",A,0),b.readyState="complete"},0)),a.yepnope=k(),a.yepnope.executeStack=h,a.yepnope.injectJs=function(a,c,d,e,i,j){var k=b.createElement("script"),l,o,e=e||B.errorTimeout;k.src=a;for(o in d)k.setAttribute(o,d[o]);c=j?h:c||f,k.onreadystatechange=k.onload=function(){!l&&g(k.readyState)&&(l=1,c(),k.onload=k.onreadystatechange=null)},m(function(){l||(l=1,c(1))},e),i?k.onload():n.parentNode.insertBefore(k,n)},a.yepnope.injectCss=function(a,c,d,e,g,i){var e=b.createElement("link"),j,c=i?h:c||f;e.href=a,e.rel="stylesheet",e.type="text/css";for(j in d)e.setAttribute(j,d[j]);g||(n.parentNode.insertBefore(e,n),m(c,0))}}(this,document),Modernizr.load=function(){yepnope.apply(window,[].slice.call(arguments,0))};
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/js/theme.js
----------------------------------------------------------------------
diff --git a/_static/js/theme.js b/_static/js/theme.js
new file mode 100644
index 0000000..48a9f06
--- /dev/null
+++ b/_static/js/theme.js
@@ -0,0 +1,153 @@
+require=(function e(t,n,r){function s(o,u){if(!n[o]){if(!t[o]){var a=typeof require=="function"&&require;if(!u&&a)return a(o,!0);if(i)return i(o,!0);var f=new Error("Cannot find module '"+o+"'");throw f.code="MODULE_NOT_FOUND",f}var l=n[o]={exports:{}};t[o][0].call(l.exports,function(e){var n=t[o][1][e];return s(n?n:e)},l,l.exports,e,t,n,r)}return n[o].exports}var i=typeof require=="function"&&require;for(var o=0;o<r.length;o++)s(r[o]);return s})({"sphinx-rtd-theme":[function(require,module,exports){
+var jQuery = (typeof(window) != 'undefined') ? window.jQuery : require('jquery');
+
+// Sphinx theme nav state
+function ThemeNav () {
+
+ var nav = {
+ navBar: null,
+ win: null,
+ winScroll: false,
+ winResize: false,
+ linkScroll: false,
+ winPosition: 0,
+ winHeight: null,
+ docHeight: null,
+ isRunning: null
+ };
+
+ nav.enable = function () {
+ var self = this;
+
+ jQuery(function ($) {
+ self.init($);
+
+ self.reset();
+ self.win.on('hashchange', self.reset);
+
+ // Set scroll monitor
+ self.win.on('scroll', function () {
+ if (!self.linkScroll) {
+ self.winScroll = true;
+ }
+ });
+ setInterval(function () { if (self.winScroll) self.onScroll(); }, 25);
+
+ // Set resize monitor
+ self.win.on('resize', function () {
+ self.winResize = true;
+ });
+ setInterval(function () { if (self.winResize) self.onResize(); }, 25);
+ self.onResize();
+ });
+ };
+
+ nav.init = function ($) {
+ var doc = $(document),
+ self = this;
+
+ this.navBar = $('div.wy-side-scroll:first');
+ this.win = $(window);
+
+ // Set up javascript UX bits
+ $(document)
+ // Shift nav in mobile when clicking the menu.
+ .on('click', "[data-toggle='wy-nav-top']", function() {
+ $("[data-toggle='wy-nav-shift']").toggleClass("shift");
+ $("[data-toggle='rst-versions']").toggleClass("shift");
+ })
+
+ // Nav menu link click operations
+ .on('click', ".wy-menu-vertical .current ul li a", function() {
+ var target = $(this);
+ // Close menu when you click a link.
+ $("[data-toggle='wy-nav-shift']").removeClass("shift");
+ $("[data-toggle='rst-versions']").toggleClass("shift");
+ // Handle dynamic display of l3 and l4 nav lists
+ self.toggleCurrent(target);
+ self.hashChange();
+ })
+ .on('click', "[data-toggle='rst-current-version']", function() {
+ $("[data-toggle='rst-versions']").toggleClass("shift-up");
+ })
+
+ // Make tables responsive
+ $("table.docutils:not(.field-list)")
+ .wrap("<div class='wy-table-responsive'></div>");
+
+ // Add expand links to all parents of nested ul
+ $('.wy-menu-vertical ul').not('.simple').siblings('a').each(function () {
+ var link = $(this);
+ expand = $('<span class="toctree-expand"></span>');
+ expand.on('click', function (ev) {
+ self.toggleCurrent(link);
+ ev.stopPropagation();
+ return false;
+ });
+ link.prepend(expand);
+ });
+ };
+
+ nav.reset = function () {
+ // Get anchor from URL and open up nested nav
+ var anchor = encodeURI(window.location.hash);
+ if (anchor) {
+ try {
+ var link = $('.wy-menu-vertical')
+ .find('[href="' + anchor + '"]');
+ $('.wy-menu-vertical li.toctree-l1 li.current')
+ .removeClass('current');
+ link.closest('li.toctree-l2').addClass('current');
+ link.closest('li.toctree-l3').addClass('current');
+ link.closest('li.toctree-l4').addClass('current');
+ }
+ catch (err) {
+ console.log("Error expanding nav for anchor", err);
+ }
+ }
+ };
+
+ nav.onScroll = function () {
+ this.winScroll = false;
+ var newWinPosition = this.win.scrollTop(),
+ winBottom = newWinPosition + this.winHeight,
+ navPosition = this.navBar.scrollTop(),
+ newNavPosition = navPosition + (newWinPosition - this.winPosition);
+ if (newWinPosition < 0 || winBottom > this.docHeight) {
+ return;
+ }
+ this.navBar.scrollTop(newNavPosition);
+ this.winPosition = newWinPosition;
+ };
+
+ nav.onResize = function () {
+ this.winResize = false;
+ this.winHeight = this.win.height();
+ this.docHeight = $(document).height();
+ };
+
+ nav.hashChange = function () {
+ this.linkScroll = true;
+ this.win.one('hashchange', function () {
+ this.linkScroll = false;
+ });
+ };
+
+ nav.toggleCurrent = function (elem) {
+ var parent_li = elem.closest('li');
+ parent_li.siblings('li.current').removeClass('current');
+ parent_li.siblings().find('li.current').removeClass('current');
+ parent_li.find('> ul li.current').removeClass('current');
+ parent_li.toggleClass('current');
+ }
+
+ return nav;
+};
+
+module.exports.ThemeNav = ThemeNav();
+
+if (typeof(window) != 'undefined') {
+ window.SphinxRtdTheme = { StickyNav: module.exports.ThemeNav };
+}
+
+},{"jquery":"jquery"}]},{},["sphinx-rtd-theme"]);
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/minus.png
----------------------------------------------------------------------
diff --git a/_static/minus.png b/_static/minus.png
new file mode 100644
index 0000000..0f22b16
Binary files /dev/null and b/_static/minus.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/plus.png
----------------------------------------------------------------------
diff --git a/_static/plus.png b/_static/plus.png
new file mode 100644
index 0000000..0cfe084
Binary files /dev/null and b/_static/plus.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/pygments.css
----------------------------------------------------------------------
diff --git a/_static/pygments.css b/_static/pygments.css
new file mode 100644
index 0000000..8213e90
--- /dev/null
+++ b/_static/pygments.css
@@ -0,0 +1,65 @@
+.highlight .hll { background-color: #ffffcc }
+.highlight { background: #eeffcc; }
+.highlight .c { color: #408090; font-style: italic } /* Comment */
+.highlight .err { border: 1px solid #FF0000 } /* Error */
+.highlight .k { color: #007020; font-weight: bold } /* Keyword */
+.highlight .o { color: #666666 } /* Operator */
+.highlight .ch { color: #408090; font-style: italic } /* Comment.Hashbang */
+.highlight .cm { color: #408090; font-style: italic } /* Comment.Multiline */
+.highlight .cp { color: #007020 } /* Comment.Preproc */
+.highlight .cpf { color: #408090; font-style: italic } /* Comment.PreprocFile */
+.highlight .c1 { color: #408090; font-style: italic } /* Comment.Single */
+.highlight .cs { color: #408090; background-color: #fff0f0 } /* Comment.Special */
+.highlight .gd { color: #A00000 } /* Generic.Deleted */
+.highlight .ge { font-style: italic } /* Generic.Emph */
+.highlight .gr { color: #FF0000 } /* Generic.Error */
+.highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */
+.highlight .gi { color: #00A000 } /* Generic.Inserted */
+.highlight .go { color: #333333 } /* Generic.Output */
+.highlight .gp { color: #c65d09; font-weight: bold } /* Generic.Prompt */
+.highlight .gs { font-weight: bold } /* Generic.Strong */
+.highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
+.highlight .gt { color: #0044DD } /* Generic.Traceback */
+.highlight .kc { color: #007020; font-weight: bold } /* Keyword.Constant */
+.highlight .kd { color: #007020; font-weight: bold } /* Keyword.Declaration */
+.highlight .kn { color: #007020; font-weight: bold } /* Keyword.Namespace */
+.highlight .kp { color: #007020 } /* Keyword.Pseudo */
+.highlight .kr { color: #007020; font-weight: bold } /* Keyword.Reserved */
+.highlight .kt { color: #902000 } /* Keyword.Type */
+.highlight .m { color: #208050 } /* Literal.Number */
+.highlight .s { color: #4070a0 } /* Literal.String */
+.highlight .na { color: #4070a0 } /* Name.Attribute */
+.highlight .nb { color: #007020 } /* Name.Builtin */
+.highlight .nc { color: #0e84b5; font-weight: bold } /* Name.Class */
+.highlight .no { color: #60add5 } /* Name.Constant */
+.highlight .nd { color: #555555; font-weight: bold } /* Name.Decorator */
+.highlight .ni { color: #d55537; font-weight: bold } /* Name.Entity */
+.highlight .ne { color: #007020 } /* Name.Exception */
+.highlight .nf { color: #06287e } /* Name.Function */
+.highlight .nl { color: #002070; font-weight: bold } /* Name.Label */
+.highlight .nn { color: #0e84b5; font-weight: bold } /* Name.Namespace */
+.highlight .nt { color: #062873; font-weight: bold } /* Name.Tag */
+.highlight .nv { color: #bb60d5 } /* Name.Variable */
+.highlight .ow { color: #007020; font-weight: bold } /* Operator.Word */
+.highlight .w { color: #bbbbbb } /* Text.Whitespace */
+.highlight .mb { color: #208050 } /* Literal.Number.Bin */
+.highlight .mf { color: #208050 } /* Literal.Number.Float */
+.highlight .mh { color: #208050 } /* Literal.Number.Hex */
+.highlight .mi { color: #208050 } /* Literal.Number.Integer */
+.highlight .mo { color: #208050 } /* Literal.Number.Oct */
+.highlight .sb { color: #4070a0 } /* Literal.String.Backtick */
+.highlight .sc { color: #4070a0 } /* Literal.String.Char */
+.highlight .sd { color: #4070a0; font-style: italic } /* Literal.String.Doc */
+.highlight .s2 { color: #4070a0 } /* Literal.String.Double */
+.highlight .se { color: #4070a0; font-weight: bold } /* Literal.String.Escape */
+.highlight .sh { color: #4070a0 } /* Literal.String.Heredoc */
+.highlight .si { color: #70a0d0; font-style: italic } /* Literal.String.Interpol */
+.highlight .sx { color: #c65d09 } /* Literal.String.Other */
+.highlight .sr { color: #235388 } /* Literal.String.Regex */
+.highlight .s1 { color: #4070a0 } /* Literal.String.Single */
+.highlight .ss { color: #517918 } /* Literal.String.Symbol */
+.highlight .bp { color: #007020 } /* Name.Builtin.Pseudo */
+.highlight .vc { color: #bb60d5 } /* Name.Variable.Class */
+.highlight .vg { color: #bb60d5 } /* Name.Variable.Global */
+.highlight .vi { color: #bb60d5 } /* Name.Variable.Instance */
+.highlight .il { color: #208050 } /* Literal.Number.Integer.Long */
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/searchtools.js
----------------------------------------------------------------------
diff --git a/_static/searchtools.js b/_static/searchtools.js
new file mode 100644
index 0000000..066857c
--- /dev/null
+++ b/_static/searchtools.js
@@ -0,0 +1,651 @@
+/*
+ * searchtools.js_t
+ * ~~~~~~~~~~~~~~~~
+ *
+ * Sphinx JavaScript utilities for the full-text search.
+ *
+ * :copyright: Copyright 2007-2016 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+
+/* Non-minified version JS is _stemmer.js if file is provided */
+/**
+ * Porter Stemmer
+ */
+var Stemmer = function() {
+
+ var step2list = {
+ ational: 'ate',
+ tional: 'tion',
+ enci: 'ence',
+ anci: 'ance',
+ izer: 'ize',
+ bli: 'ble',
+ alli: 'al',
+ entli: 'ent',
+ eli: 'e',
+ ousli: 'ous',
+ ization: 'ize',
+ ation: 'ate',
+ ator: 'ate',
+ alism: 'al',
+ iveness: 'ive',
+ fulness: 'ful',
+ ousness: 'ous',
+ aliti: 'al',
+ iviti: 'ive',
+ biliti: 'ble',
+ logi: 'log'
+ };
+
+ var step3list = {
+ icate: 'ic',
+ ative: '',
+ alize: 'al',
+ iciti: 'ic',
+ ical: 'ic',
+ ful: '',
+ ness: ''
+ };
+
+ var c = "[^aeiou]"; // consonant
+ var v = "[aeiouy]"; // vowel
+ var C = c + "[^aeiouy]*"; // consonant sequence
+ var V = v + "[aeiou]*"; // vowel sequence
+
+ var mgr0 = "^(" + C + ")?" + V + C; // [C]VC... is m>0
+ var meq1 = "^(" + C + ")?" + V + C + "(" + V + ")?$"; // [C]VC[V] is m=1
+ var mgr1 = "^(" + C + ")?" + V + C + V + C; // [C]VCVC... is m>1
+ var s_v = "^(" + C + ")?" + v; // vowel in stem
+
+ this.stemWord = function (w) {
+ var stem;
+ var suffix;
+ var firstch;
+ var origword = w;
+
+ if (w.length < 3)
+ return w;
+
+ var re;
+ var re2;
+ var re3;
+ var re4;
+
+ firstch = w.substr(0,1);
+ if (firstch == "y")
+ w = firstch.toUpperCase() + w.substr(1);
+
+ // Step 1a
+ re = /^(.+?)(ss|i)es$/;
+ re2 = /^(.+?)([^s])s$/;
+
+ if (re.test(w))
+ w = w.replace(re,"$1$2");
+ else if (re2.test(w))
+ w = w.replace(re2,"$1$2");
+
+ // Step 1b
+ re = /^(.+?)eed$/;
+ re2 = /^(.+?)(ed|ing)$/;
+ if (re.test(w)) {
+ var fp = re.exec(w);
+ re = new RegExp(mgr0);
+ if (re.test(fp[1])) {
+ re = /.$/;
+ w = w.replace(re,"");
+ }
+ }
+ else if (re2.test(w)) {
+ var fp = re2.exec(w);
+ stem = fp[1];
+ re2 = new RegExp(s_v);
+ if (re2.test(stem)) {
+ w = stem;
+ re2 = /(at|bl|iz)$/;
+ re3 = new RegExp("([^aeiouylsz])\\1$");
+ re4 = new RegExp("^" + C + v + "[^aeiouwxy]$");
+ if (re2.test(w))
+ w = w + "e";
+ else if (re3.test(w)) {
+ re = /.$/;
+ w = w.replace(re,"");
+ }
+ else if (re4.test(w))
+ w = w + "e";
+ }
+ }
+
+ // Step 1c
+ re = /^(.+?)y$/;
+ if (re.test(w)) {
+ var fp = re.exec(w);
+ stem = fp[1];
+ re = new RegExp(s_v);
+ if (re.test(stem))
+ w = stem + "i";
+ }
+
+ // Step 2
+ re = /^(.+?)(ational|tional|enci|anci|izer|bli|alli|entli|eli|ousli|ization|ation|ator|alism|iveness|fulness|ousness|aliti|iviti|biliti|logi)$/;
+ if (re.test(w)) {
+ var fp = re.exec(w);
+ stem = fp[1];
+ suffix = fp[2];
+ re = new RegExp(mgr0);
+ if (re.test(stem))
+ w = stem + step2list[suffix];
+ }
+
+ // Step 3
+ re = /^(.+?)(icate|ative|alize|iciti|ical|ful|ness)$/;
+ if (re.test(w)) {
+ var fp = re.exec(w);
+ stem = fp[1];
+ suffix = fp[2];
+ re = new RegExp(mgr0);
+ if (re.test(stem))
+ w = stem + step3list[suffix];
+ }
+
+ // Step 4
+ re = /^(.+?)(al|ance|ence|er|ic|able|ible|ant|ement|ment|ent|ou|ism|ate|iti|ous|ive|ize)$/;
+ re2 = /^(.+?)(s|t)(ion)$/;
+ if (re.test(w)) {
+ var fp = re.exec(w);
+ stem = fp[1];
+ re = new RegExp(mgr1);
+ if (re.test(stem))
+ w = stem;
+ }
+ else if (re2.test(w)) {
+ var fp = re2.exec(w);
+ stem = fp[1] + fp[2];
+ re2 = new RegExp(mgr1);
+ if (re2.test(stem))
+ w = stem;
+ }
+
+ // Step 5
+ re = /^(.+?)e$/;
+ if (re.test(w)) {
+ var fp = re.exec(w);
+ stem = fp[1];
+ re = new RegExp(mgr1);
+ re2 = new RegExp(meq1);
+ re3 = new RegExp("^" + C + v + "[^aeiouwxy]$");
+ if (re.test(stem) || (re2.test(stem) && !(re3.test(stem))))
+ w = stem;
+ }
+ re = /ll$/;
+ re2 = new RegExp(mgr1);
+ if (re.test(w) && re2.test(w)) {
+ re = /.$/;
+ w = w.replace(re,"");
+ }
+
+ // and turn initial Y back to y
+ if (firstch == "y")
+ w = firstch.toLowerCase() + w.substr(1);
+ return w;
+ }
+}
+
+
+
+/**
+ * Simple result scoring code.
+ */
+var Scorer = {
+ // Implement the following function to further tweak the score for each result
+ // The function takes a result array [filename, title, anchor, descr, score]
+ // and returns the new score.
+ /*
+ score: function(result) {
+ return result[4];
+ },
+ */
+
+ // query matches the full name of an object
+ objNameMatch: 11,
+ // or matches in the last dotted part of the object name
+ objPartialMatch: 6,
+ // Additive scores depending on the priority of the object
+ objPrio: {0: 15, // used to be importantResults
+ 1: 5, // used to be objectResults
+ 2: -5}, // used to be unimportantResults
+ // Used when the priority is not in the mapping.
+ objPrioDefault: 0,
+
+ // query found in title
+ title: 15,
+ // query found in terms
+ term: 5
+};
+
+
+/**
+ * Search Module
+ */
+var Search = {
+
+ _index : null,
+ _queued_query : null,
+ _pulse_status : -1,
+
+ init : function() {
+ var params = $.getQueryParameters();
+ if (params.q) {
+ var query = params.q[0];
+ $('input[name="q"]')[0].value = query;
+ this.performSearch(query);
+ }
+ },
+
+ loadIndex : function(url) {
+ $.ajax({type: "GET", url: url, data: null,
+ dataType: "script", cache: true,
+ complete: function(jqxhr, textstatus) {
+ if (textstatus != "success") {
+ document.getElementById("searchindexloader").src = url;
+ }
+ }});
+ },
+
+ setIndex : function(index) {
+ var q;
+ this._index = index;
+ if ((q = this._queued_query) !== null) {
+ this._queued_query = null;
+ Search.query(q);
+ }
+ },
+
+ hasIndex : function() {
+ return this._index !== null;
+ },
+
+ deferQuery : function(query) {
+ this._queued_query = query;
+ },
+
+ stopPulse : function() {
+ this._pulse_status = 0;
+ },
+
+ startPulse : function() {
+ if (this._pulse_status >= 0)
+ return;
+ function pulse() {
+ var i;
+ Search._pulse_status = (Search._pulse_status + 1) % 4;
+ var dotString = '';
+ for (i = 0; i < Search._pulse_status; i++)
+ dotString += '.';
+ Search.dots.text(dotString);
+ if (Search._pulse_status > -1)
+ window.setTimeout(pulse, 500);
+ }
+ pulse();
+ },
+
+ /**
+ * perform a search for something (or wait until index is loaded)
+ */
+ performSearch : function(query) {
+ // create the required interface elements
+ this.out = $('#search-results');
+ this.title = $('<h2>' + _('Searching') + '</h2>').appendTo(this.out);
+ this.dots = $('<span></span>').appendTo(this.title);
+ this.status = $('<p style="display: none"></p>').appendTo(this.out);
+ this.output = $('<ul class="search"/>').appendTo(this.out);
+
+ $('#search-progress').text(_('Preparing search...'));
+ this.startPulse();
+
+ // index already loaded, the browser was quick!
+ if (this.hasIndex())
+ this.query(query);
+ else
+ this.deferQuery(query);
+ },
+
+ /**
+ * execute search (requires search index to be loaded)
+ */
+ query : function(query) {
+ var i;
+ var stopwords = ["a","and","are","as","at","be","but","by","for","if","in","into","is","it","near","no","not","of","on","or","such","that","the","their","then","there","these","they","this","to","was","will","with"];
+
+ // stem the searchterms and add them to the correct list
+ var stemmer = new Stemmer();
+ var searchterms = [];
+ var excluded = [];
+ var hlterms = [];
+ var tmp = query.split(/\s+/);
+ var objectterms = [];
+ for (i = 0; i < tmp.length; i++) {
+ if (tmp[i] !== "") {
+ objectterms.push(tmp[i].toLowerCase());
+ }
+
+ if ($u.indexOf(stopwords, tmp[i].toLowerCase()) != -1 || tmp[i].match(/^\d+$/) ||
+ tmp[i] === "") {
+ // skip this "word"
+ continue;
+ }
+ // stem the word
+ var word = stemmer.stemWord(tmp[i].toLowerCase());
+ var toAppend;
+ // select the correct list
+ if (word[0] == '-') {
+ toAppend = excluded;
+ word = word.substr(1);
+ }
+ else {
+ toAppend = searchterms;
+ hlterms.push(tmp[i].toLowerCase());
+ }
+ // only add if not already in the list
+ if (!$u.contains(toAppend, word))
+ toAppend.push(word);
+ }
+ var highlightstring = '?highlight=' + $.urlencode(hlterms.join(" "));
+
+ // console.debug('SEARCH: searching for:');
+ // console.info('required: ', searchterms);
+ // console.info('excluded: ', excluded);
+
+ // prepare search
+ var terms = this._index.terms;
+ var titleterms = this._index.titleterms;
+
+ // array of [filename, title, anchor, descr, score]
+ var results = [];
+ $('#search-progress').empty();
+
+ // lookup as object
+ for (i = 0; i < objectterms.length; i++) {
+ var others = [].concat(objectterms.slice(0, i),
+ objectterms.slice(i+1, objectterms.length));
+ results = results.concat(this.performObjectSearch(objectterms[i], others));
+ }
+
+ // lookup as search terms in fulltext
+ results = results.concat(this.performTermsSearch(searchterms, excluded, terms, titleterms));
+
+ // let the scorer override scores with a custom scoring function
+ if (Scorer.score) {
+ for (i = 0; i < results.length; i++)
+ results[i][4] = Scorer.score(results[i]);
+ }
+
+ // now sort the results by score (in opposite order of appearance, since the
+ // display function below uses pop() to retrieve items) and then
+ // alphabetically
+ results.sort(function(a, b) {
+ var left = a[4];
+ var right = b[4];
+ if (left > right) {
+ return 1;
+ } else if (left < right) {
+ return -1;
+ } else {
+ // same score: sort alphabetically
+ left = a[1].toLowerCase();
+ right = b[1].toLowerCase();
+ return (left > right) ? -1 : ((left < right) ? 1 : 0);
+ }
+ });
+
+ // for debugging
+ //Search.lastresults = results.slice(); // a copy
+ //console.info('search results:', Search.lastresults);
+
+ // print the results
+ var resultCount = results.length;
+ function displayNextItem() {
+ // results left, load the summary and display it
+ if (results.length) {
+ var item = results.pop();
+ var listItem = $('<li style="display:none"></li>');
+ if (DOCUMENTATION_OPTIONS.FILE_SUFFIX === '') {
+ // dirhtml builder
+ var dirname = item[0] + '/';
+ if (dirname.match(/\/index\/$/)) {
+ dirname = dirname.substring(0, dirname.length-6);
+ } else if (dirname == 'index/') {
+ dirname = '';
+ }
+ listItem.append($('<a/>').attr('href',
+ DOCUMENTATION_OPTIONS.URL_ROOT + dirname +
+ highlightstring + item[2]).html(item[1]));
+ } else {
+ // normal html builders
+ listItem.append($('<a/>').attr('href',
+ item[0] + DOCUMENTATION_OPTIONS.FILE_SUFFIX +
+ highlightstring + item[2]).html(item[1]));
+ }
+ if (item[3]) {
+ listItem.append($('<span> (' + item[3] + ')</span>'));
+ Search.output.append(listItem);
+ listItem.slideDown(5, function() {
+ displayNextItem();
+ });
+ } else if (DOCUMENTATION_OPTIONS.HAS_SOURCE) {
+ $.ajax({url: DOCUMENTATION_OPTIONS.URL_ROOT + '_sources/' + item[0] + '.txt',
+ dataType: "text",
+ complete: function(jqxhr, textstatus) {
+ var data = jqxhr.responseText;
+ if (data !== '' && data !== undefined) {
+ listItem.append(Search.makeSearchSummary(data, searchterms, hlterms));
+ }
+ Search.output.append(listItem);
+ listItem.slideDown(5, function() {
+ displayNextItem();
+ });
+ }});
+ } else {
+ // no source available, just display title
+ Search.output.append(listItem);
+ listItem.slideDown(5, function() {
+ displayNextItem();
+ });
+ }
+ }
+ // search finished, update title and status message
+ else {
+ Search.stopPulse();
+ Search.title.text(_('Search Results'));
+ if (!resultCount)
+ Search.status.text(_('Your search did not match any documents. Please make sure that all words are spelled correctly and that you\'ve selected enough categories.'));
+ else
+ Search.status.text(_('Search finished, found %s page(s) matching the search query.').replace('%s', resultCount));
+ Search.status.fadeIn(500);
+ }
+ }
+ displayNextItem();
+ },
+
+ /**
+ * search for object names
+ */
+ performObjectSearch : function(object, otherterms) {
+ var filenames = this._index.filenames;
+ var objects = this._index.objects;
+ var objnames = this._index.objnames;
+ var titles = this._index.titles;
+
+ var i;
+ var results = [];
+
+ for (var prefix in objects) {
+ for (var name in objects[prefix]) {
+ var fullname = (prefix ? prefix + '.' : '') + name;
+ if (fullname.toLowerCase().indexOf(object) > -1) {
+ var score = 0;
+ var parts = fullname.split('.');
+ // check for different match types: exact matches of full name or
+ // "last name" (i.e. last dotted part)
+ if (fullname == object || parts[parts.length - 1] == object) {
+ score += Scorer.objNameMatch;
+ // matches in last name
+ } else if (parts[parts.length - 1].indexOf(object) > -1) {
+ score += Scorer.objPartialMatch;
+ }
+ var match = objects[prefix][name];
+ var objname = objnames[match[1]][2];
+ var title = titles[match[0]];
+ // If more than one term searched for, we require other words to be
+ // found in the name/title/description
+ if (otherterms.length > 0) {
+ var haystack = (prefix + ' ' + name + ' ' +
+ objname + ' ' + title).toLowerCase();
+ var allfound = true;
+ for (i = 0; i < otherterms.length; i++) {
+ if (haystack.indexOf(otherterms[i]) == -1) {
+ allfound = false;
+ break;
+ }
+ }
+ if (!allfound) {
+ continue;
+ }
+ }
+ var descr = objname + _(', in ') + title;
+
+ var anchor = match[3];
+ if (anchor === '')
+ anchor = fullname;
+ else if (anchor == '-')
+ anchor = objnames[match[1]][1] + '-' + fullname;
+ // add custom score for some objects according to scorer
+ if (Scorer.objPrio.hasOwnProperty(match[2])) {
+ score += Scorer.objPrio[match[2]];
+ } else {
+ score += Scorer.objPrioDefault;
+ }
+ results.push([filenames[match[0]], fullname, '#'+anchor, descr, score]);
+ }
+ }
+ }
+
+ return results;
+ },
+
+ /**
+ * search for full-text terms in the index
+ */
+ performTermsSearch : function(searchterms, excluded, terms, titleterms) {
+ var filenames = this._index.filenames;
+ var titles = this._index.titles;
+
+ var i, j, file;
+ var fileMap = {};
+ var scoreMap = {};
+ var results = [];
+
+ // perform the search on the required terms
+ for (i = 0; i < searchterms.length; i++) {
+ var word = searchterms[i];
+ var files = [];
+ var _o = [
+ {files: terms[word], score: Scorer.term},
+ {files: titleterms[word], score: Scorer.title}
+ ];
+
+ // no match but word was a required one
+ if ($u.every(_o, function(o){return o.files === undefined;})) {
+ break;
+ }
+ // found search word in contents
+ $u.each(_o, function(o) {
+ var _files = o.files;
+ if (_files === undefined)
+ return
+
+ if (_files.length === undefined)
+ _files = [_files];
+ files = files.concat(_files);
+
+ // set score for the word in each file to Scorer.term
+ for (j = 0; j < _files.length; j++) {
+ file = _files[j];
+ if (!(file in scoreMap))
+ scoreMap[file] = {}
+ scoreMap[file][word] = o.score;
+ }
+ });
+
+ // create the mapping
+ for (j = 0; j < files.length; j++) {
+ file = files[j];
+ if (file in fileMap)
+ fileMap[file].push(word);
+ else
+ fileMap[file] = [word];
+ }
+ }
+
+ // now check if the files don't contain excluded terms
+ for (file in fileMap) {
+ var valid = true;
+
+ // check if all requirements are matched
+ if (fileMap[file].length != searchterms.length)
+ continue;
+
+ // ensure that none of the excluded terms is in the search result
+ for (i = 0; i < excluded.length; i++) {
+ if (terms[excluded[i]] == file ||
+ titleterms[excluded[i]] == file ||
+ $u.contains(terms[excluded[i]] || [], file) ||
+ $u.contains(titleterms[excluded[i]] || [], file)) {
+ valid = false;
+ break;
+ }
+ }
+
+ // if we have still a valid result we can add it to the result list
+ if (valid) {
+ // select one (max) score for the file.
+ // for better ranking, we should calculate ranking by using words statistics like basic tf-idf...
+ var score = $u.max($u.map(fileMap[file], function(w){return scoreMap[file][w]}));
+ results.push([filenames[file], titles[file], '', null, score]);
+ }
+ }
+ return results;
+ },
+
+ /**
+ * helper function to return a node containing the
+ * search summary for a given text. keywords is a list
+ * of stemmed words, hlwords is the list of normal, unstemmed
+ * words. the first one is used to find the occurrence, the
+ * latter for highlighting it.
+ */
+ makeSearchSummary : function(text, keywords, hlwords) {
+ var textLower = text.toLowerCase();
+ var start = 0;
+ $.each(keywords, function() {
+ var i = textLower.indexOf(this.toLowerCase());
+ if (i > -1)
+ start = i;
+ });
+ start = Math.max(start - 120, 0);
+ var excerpt = ((start > 0) ? '...' : '') +
+ $.trim(text.substr(start, 240)) +
+ ((start + 240 - text.length) ? '...' : '');
+ var rv = $('<div class="context"></div>').text(excerpt);
+ $.each(hlwords, function() {
+ rv = rv.highlightText(this, 'highlighted');
+ });
+ return rv;
+ }
+};
+
+$(document).ready(function() {
+ Search.init();
+});
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/underscore-1.3.1.js
----------------------------------------------------------------------
diff --git a/_static/underscore-1.3.1.js b/_static/underscore-1.3.1.js
new file mode 100644
index 0000000..208d4cd
--- /dev/null
+++ b/_static/underscore-1.3.1.js
@@ -0,0 +1,999 @@
+// Underscore.js 1.3.1
+// (c) 2009-2012 Jeremy Ashkenas, DocumentCloud Inc.
+// Underscore is freely distributable under the MIT license.
+// Portions of Underscore are inspired or borrowed from Prototype,
+// Oliver Steele's Functional, and John Resig's Micro-Templating.
+// For all details and documentation:
+// http://documentcloud.github.com/underscore
+
+(function() {
+
+ // Baseline setup
+ // --------------
+
+ // Establish the root object, `window` in the browser, or `global` on the server.
+ var root = this;
+
+ // Save the previous value of the `_` variable.
+ var previousUnderscore = root._;
+
+ // Establish the object that gets returned to break out of a loop iteration.
+ var breaker = {};
+
+ // Save bytes in the minified (but not gzipped) version:
+ var ArrayProto = Array.prototype, ObjProto = Object.prototype, FuncProto = Function.prototype;
+
+ // Create quick reference variables for speed access to core prototypes.
+ var slice = ArrayProto.slice,
+ unshift = ArrayProto.unshift,
+ toString = ObjProto.toString,
+ hasOwnProperty = ObjProto.hasOwnProperty;
+
+ // All **ECMAScript 5** native function implementations that we hope to use
+ // are declared here.
+ var
+ nativeForEach = ArrayProto.forEach,
+ nativeMap = ArrayProto.map,
+ nativeReduce = ArrayProto.reduce,
+ nativeReduceRight = ArrayProto.reduceRight,
+ nativeFilter = ArrayProto.filter,
+ nativeEvery = ArrayProto.every,
+ nativeSome = ArrayProto.some,
+ nativeIndexOf = ArrayProto.indexOf,
+ nativeLastIndexOf = ArrayProto.lastIndexOf,
+ nativeIsArray = Array.isArray,
+ nativeKeys = Object.keys,
+ nativeBind = FuncProto.bind;
+
+ // Create a safe reference to the Underscore object for use below.
+ var _ = function(obj) { return new wrapper(obj); };
+
+ // Export the Underscore object for **Node.js**, with
+ // backwards-compatibility for the old `require()` API. If we're in
+ // the browser, add `_` as a global object via a string identifier,
+ // for Closure Compiler "advanced" mode.
+ if (typeof exports !== 'undefined') {
+ if (typeof module !== 'undefined' && module.exports) {
+ exports = module.exports = _;
+ }
+ exports._ = _;
+ } else {
+ root['_'] = _;
+ }
+
+ // Current version.
+ _.VERSION = '1.3.1';
+
+ // Collection Functions
+ // --------------------
+
+ // The cornerstone, an `each` implementation, aka `forEach`.
+ // Handles objects with the built-in `forEach`, arrays, and raw objects.
+ // Delegates to **ECMAScript 5**'s native `forEach` if available.
+ var each = _.each = _.forEach = function(obj, iterator, context) {
+ if (obj == null) return;
+ if (nativeForEach && obj.forEach === nativeForEach) {
+ obj.forEach(iterator, context);
+ } else if (obj.length === +obj.length) {
+ for (var i = 0, l = obj.length; i < l; i++) {
+ if (i in obj && iterator.call(context, obj[i], i, obj) === breaker) return;
+ }
+ } else {
+ for (var key in obj) {
+ if (_.has(obj, key)) {
+ if (iterator.call(context, obj[key], key, obj) === breaker) return;
+ }
+ }
+ }
+ };
+
+ // Return the results of applying the iterator to each element.
+ // Delegates to **ECMAScript 5**'s native `map` if available.
+ _.map = _.collect = function(obj, iterator, context) {
+ var results = [];
+ if (obj == null) return results;
+ if (nativeMap && obj.map === nativeMap) return obj.map(iterator, context);
+ each(obj, function(value, index, list) {
+ results[results.length] = iterator.call(context, value, index, list);
+ });
+ if (obj.length === +obj.length) results.length = obj.length;
+ return results;
+ };
+
+ // **Reduce** builds up a single result from a list of values, aka `inject`,
+ // or `foldl`. Delegates to **ECMAScript 5**'s native `reduce` if available.
+ _.reduce = _.foldl = _.inject = function(obj, iterator, memo, context) {
+ var initial = arguments.length > 2;
+ if (obj == null) obj = [];
+ if (nativeReduce && obj.reduce === nativeReduce) {
+ if (context) iterator = _.bind(iterator, context);
+ return initial ? obj.reduce(iterator, memo) : obj.reduce(iterator);
+ }
+ each(obj, function(value, index, list) {
+ if (!initial) {
+ memo = value;
+ initial = true;
+ } else {
+ memo = iterator.call(context, memo, value, index, list);
+ }
+ });
+ if (!initial) throw new TypeError('Reduce of empty array with no initial value');
+ return memo;
+ };
+
+ // The right-associative version of reduce, also known as `foldr`.
+ // Delegates to **ECMAScript 5**'s native `reduceRight` if available.
+ _.reduceRight = _.foldr = function(obj, iterator, memo, context) {
+ var initial = arguments.length > 2;
+ if (obj == null) obj = [];
+ if (nativeReduceRight && obj.reduceRight === nativeReduceRight) {
+ if (context) iterator = _.bind(iterator, context);
+ return initial ? obj.reduceRight(iterator, memo) : obj.reduceRight(iterator);
+ }
+ var reversed = _.toArray(obj).reverse();
+ if (context && !initial) iterator = _.bind(iterator, context);
+ return initial ? _.reduce(reversed, iterator, memo, context) : _.reduce(reversed, iterator);
+ };
+
+ // Return the first value which passes a truth test. Aliased as `detect`.
+ _.find = _.detect = function(obj, iterator, context) {
+ var result;
+ any(obj, function(value, index, list) {
+ if (iterator.call(context, value, index, list)) {
+ result = value;
+ return true;
+ }
+ });
+ return result;
+ };
+
+ // Return all the elements that pass a truth test.
+ // Delegates to **ECMAScript 5**'s native `filter` if available.
+ // Aliased as `select`.
+ _.filter = _.select = function(obj, iterator, context) {
+ var results = [];
+ if (obj == null) return results;
+ if (nativeFilter && obj.filter === nativeFilter) return obj.filter(iterator, context);
+ each(obj, function(value, index, list) {
+ if (iterator.call(context, value, index, list)) results[results.length] = value;
+ });
+ return results;
+ };
+
+ // Return all the elements for which a truth test fails.
+ _.reject = function(obj, iterator, context) {
+ var results = [];
+ if (obj == null) return results;
+ each(obj, function(value, index, list) {
+ if (!iterator.call(context, value, index, list)) results[results.length] = value;
+ });
+ return results;
+ };
+
+ // Determine whether all of the elements match a truth test.
+ // Delegates to **ECMAScript 5**'s native `every` if available.
+ // Aliased as `all`.
+ _.every = _.all = function(obj, iterator, context) {
+ var result = true;
+ if (obj == null) return result;
+ if (nativeEvery && obj.every === nativeEvery) return obj.every(iterator, context);
+ each(obj, function(value, index, list) {
+ if (!(result = result && iterator.call(context, value, index, list))) return breaker;
+ });
+ return result;
+ };
+
+ // Determine if at least one element in the object matches a truth test.
+ // Delegates to **ECMAScript 5**'s native `some` if available.
+ // Aliased as `any`.
+ var any = _.some = _.any = function(obj, iterator, context) {
+ iterator || (iterator = _.identity);
+ var result = false;
+ if (obj == null) return result;
+ if (nativeSome && obj.some === nativeSome) return obj.some(iterator, context);
+ each(obj, function(value, index, list) {
+ if (result || (result = iterator.call(context, value, index, list))) return breaker;
+ });
+ return !!result;
+ };
+
+ // Determine if a given value is included in the array or object using `===`.
+ // Aliased as `contains`.
+ _.include = _.contains = function(obj, target) {
+ var found = false;
+ if (obj == null) return found;
+ if (nativeIndexOf && obj.indexOf === nativeIndexOf) return obj.indexOf(target) != -1;
+ found = any(obj, function(value) {
+ return value === target;
+ });
+ return found;
+ };
+
+ // Invoke a method (with arguments) on every item in a collection.
+ _.invoke = function(obj, method) {
+ var args = slice.call(arguments, 2);
+ return _.map(obj, function(value) {
+ return (_.isFunction(method) ? method || value : value[method]).apply(value, args);
+ });
+ };
+
+ // Convenience version of a common use case of `map`: fetching a property.
+ _.pluck = function(obj, key) {
+ return _.map(obj, function(value){ return value[key]; });
+ };
+
+ // Return the maximum element or (element-based computation).
+ _.max = function(obj, iterator, context) {
+ if (!iterator && _.isArray(obj)) return Math.max.apply(Math, obj);
+ if (!iterator && _.isEmpty(obj)) return -Infinity;
+ var result = {computed : -Infinity};
+ each(obj, function(value, index, list) {
+ var computed = iterator ? iterator.call(context, value, index, list) : value;
+ computed >= result.computed && (result = {value : value, computed : computed});
+ });
+ return result.value;
+ };
+
+ // Return the minimum element (or element-based computation).
+ _.min = function(obj, iterator, context) {
+ if (!iterator && _.isArray(obj)) return Math.min.apply(Math, obj);
+ if (!iterator && _.isEmpty(obj)) return Infinity;
+ var result = {computed : Infinity};
+ each(obj, function(value, index, list) {
+ var computed = iterator ? iterator.call(context, value, index, list) : value;
+ computed < result.computed && (result = {value : value, computed : computed});
+ });
+ return result.value;
+ };
+
+ // Shuffle an array.
+ _.shuffle = function(obj) {
+ var shuffled = [], rand;
+ each(obj, function(value, index, list) {
+ if (index == 0) {
+ shuffled[0] = value;
+ } else {
+ rand = Math.floor(Math.random() * (index + 1));
+ shuffled[index] = shuffled[rand];
+ shuffled[rand] = value;
+ }
+ });
+ return shuffled;
+ };
+
+ // Sort the object's values by a criterion produced by an iterator.
+ _.sortBy = function(obj, iterator, context) {
+ return _.pluck(_.map(obj, function(value, index, list) {
+ return {
+ value : value,
+ criteria : iterator.call(context, value, index, list)
+ };
+ }).sort(function(left, right) {
+ var a = left.criteria, b = right.criteria;
+ return a < b ? -1 : a > b ? 1 : 0;
+ }), 'value');
+ };
+
+ // Groups the object's values by a criterion. Pass either a string attribute
+ // to group by, or a function that returns the criterion.
+ _.groupBy = function(obj, val) {
+ var result = {};
+ var iterator = _.isFunction(val) ? val : function(obj) { return obj[val]; };
+ each(obj, function(value, index) {
+ var key = iterator(value, index);
+ (result[key] || (result[key] = [])).push(value);
+ });
+ return result;
+ };
+
+ // Use a comparator function to figure out at what index an object should
+ // be inserted so as to maintain order. Uses binary search.
+ _.sortedIndex = function(array, obj, iterator) {
+ iterator || (iterator = _.identity);
+ var low = 0, high = array.length;
+ while (low < high) {
+ var mid = (low + high) >> 1;
+ iterator(array[mid]) < iterator(obj) ? low = mid + 1 : high = mid;
+ }
+ return low;
+ };
+
+ // Safely convert anything iterable into a real, live array.
+ _.toArray = function(iterable) {
+ if (!iterable) return [];
+ if (iterable.toArray) return iterable.toArray();
+ if (_.isArray(iterable)) return slice.call(iterable);
+ if (_.isArguments(iterable)) return slice.call(iterable);
+ return _.values(iterable);
+ };
+
+ // Return the number of elements in an object.
+ _.size = function(obj) {
+ return _.toArray(obj).length;
+ };
+
+ // Array Functions
+ // ---------------
+
+ // Get the first element of an array. Passing **n** will return the first N
+ // values in the array. Aliased as `head`. The **guard** check allows it to work
+ // with `_.map`.
+ _.first = _.head = function(array, n, guard) {
+ return (n != null) && !guard ? slice.call(array, 0, n) : array[0];
+ };
+
+ // Returns everything but the last entry of the array. Especcialy useful on
+ // the arguments object. Passing **n** will return all the values in
+ // the array, excluding the last N. The **guard** check allows it to work with
+ // `_.map`.
+ _.initial = function(array, n, guard) {
+ return slice.call(array, 0, array.length - ((n == null) || guard ? 1 : n));
+ };
+
+ // Get the last element of an array. Passing **n** will return the last N
+ // values in the array. The **guard** check allows it to work with `_.map`.
+ _.last = function(array, n, guard) {
+ if ((n != null) && !guard) {
+ return slice.call(array, Math.max(array.length - n, 0));
+ } else {
+ return array[array.length - 1];
+ }
+ };
+
+ // Returns everything but the first entry of the array. Aliased as `tail`.
+ // Especially useful on the arguments object. Passing an **index** will return
+ // the rest of the values in the array from that index onward. The **guard**
+ // check allows it to work with `_.map`.
+ _.rest = _.tail = function(array, index, guard) {
+ return slice.call(array, (index == null) || guard ? 1 : index);
+ };
+
+ // Trim out all falsy values from an array.
+ _.compact = function(array) {
+ return _.filter(array, function(value){ return !!value; });
+ };
+
+ // Return a completely flattened version of an array.
+ _.flatten = function(array, shallow) {
+ return _.reduce(array, function(memo, value) {
+ if (_.isArray(value)) return memo.concat(shallow ? value : _.flatten(value));
+ memo[memo.length] = value;
+ return memo;
+ }, []);
+ };
+
+ // Return a version of the array that does not contain the specified value(s).
+ _.without = function(array) {
+ return _.difference(array, slice.call(arguments, 1));
+ };
+
+ // Produce a duplicate-free version of the array. If the array has already
+ // been sorted, you have the option of using a faster algorithm.
+ // Aliased as `unique`.
+ _.uniq = _.unique = function(array, isSorted, iterator) {
+ var initial = iterator ? _.map(array, iterator) : array;
+ var result = [];
+ _.reduce(initial, function(memo, el, i) {
+ if (0 == i || (isSorted === true ? _.last(memo) != el : !_.include(memo, el))) {
+ memo[memo.length] = el;
+ result[result.length] = array[i];
+ }
+ return memo;
+ }, []);
+ return result;
+ };
+
+ // Produce an array that contains the union: each distinct element from all of
+ // the passed-in arrays.
+ _.union = function() {
+ return _.uniq(_.flatten(arguments, true));
+ };
+
+ // Produce an array that contains every item shared between all the
+ // passed-in arrays. (Aliased as "intersect" for back-compat.)
+ _.intersection = _.intersect = function(array) {
+ var rest = slice.call(arguments, 1);
+ return _.filter(_.uniq(array), function(item) {
+ return _.every(rest, function(other) {
+ return _.indexOf(other, item) >= 0;
+ });
+ });
+ };
+
+ // Take the difference between one array and a number of other arrays.
+ // Only the elements present in just the first array will remain.
+ _.difference = function(array) {
+ var rest = _.flatten(slice.call(arguments, 1));
+ return _.filter(array, function(value){ return !_.include(rest, value); });
+ };
+
+ // Zip together multiple lists into a single array -- elements that share
+ // an index go together.
+ _.zip = function() {
+ var args = slice.call(arguments);
+ var length = _.max(_.pluck(args, 'length'));
+ var results = new Array(length);
+ for (var i = 0; i < length; i++) results[i] = _.pluck(args, "" + i);
+ return results;
+ };
+
+ // If the browser doesn't supply us with indexOf (I'm looking at you, **MSIE**),
+ // we need this function. Return the position of the first occurrence of an
+ // item in an array, or -1 if the item is not included in the array.
+ // Delegates to **ECMAScript 5**'s native `indexOf` if available.
+ // If the array is large and already in sort order, pass `true`
+ // for **isSorted** to use binary search.
+ _.indexOf = function(array, item, isSorted) {
+ if (array == null) return -1;
+ var i, l;
+ if (isSorted) {
+ i = _.sortedIndex(array, item);
+ return array[i] === item ? i : -1;
+ }
+ if (nativeIndexOf && array.indexOf === nativeIndexOf) return array.indexOf(item);
+ for (i = 0, l = array.length; i < l; i++) if (i in array && array[i] === item) return i;
+ return -1;
+ };
+
+ // Delegates to **ECMAScript 5**'s native `lastIndexOf` if available.
+ _.lastIndexOf = function(array, item) {
+ if (array == null) return -1;
+ if (nativeLastIndexOf && array.lastIndexOf === nativeLastIndexOf) return array.lastIndexOf(item);
+ var i = array.length;
+ while (i--) if (i in array && array[i] === item) return i;
+ return -1;
+ };
+
+ // Generate an integer Array containing an arithmetic progression. A port of
+ // the native Python `range()` function. See
+ // [the Python documentation](http://docs.python.org/library/functions.html#range).
+ _.range = function(start, stop, step) {
+ if (arguments.length <= 1) {
+ stop = start || 0;
+ start = 0;
+ }
+ step = arguments[2] || 1;
+
+ var len = Math.max(Math.ceil((stop - start) / step), 0);
+ var idx = 0;
+ var range = new Array(len);
+
+ while(idx < len) {
+ range[idx++] = start;
+ start += step;
+ }
+
+ return range;
+ };
+
+ // Function (ahem) Functions
+ // ------------------
+
+ // Reusable constructor function for prototype setting.
+ var ctor = function(){};
+
+ // Create a function bound to a given object (assigning `this`, and arguments,
+ // optionally). Binding with arguments is also known as `curry`.
+ // Delegates to **ECMAScript 5**'s native `Function.bind` if available.
+ // We check for `func.bind` first, to fail fast when `func` is undefined.
+ _.bind = function bind(func, context) {
+ var bound, args;
+ if (func.bind === nativeBind && nativeBind) return nativeBind.apply(func, slice.call(arguments, 1));
+ if (!_.isFunction(func)) throw new TypeError;
+ args = slice.call(arguments, 2);
+ return bound = function() {
+ if (!(this instanceof bound)) return func.apply(context, args.concat(slice.call(arguments)));
+ ctor.prototype = func.prototype;
+ var self = new ctor;
+ var result = func.apply(self, args.concat(slice.call(arguments)));
+ if (Object(result) === result) return result;
+ return self;
+ };
+ };
+
+ // Bind all of an object's methods to that object. Useful for ensuring that
+ // all callbacks defined on an object belong to it.
+ _.bindAll = function(obj) {
+ var funcs = slice.call(arguments, 1);
+ if (funcs.length == 0) funcs = _.functions(obj);
+ each(funcs, function(f) { obj[f] = _.bind(obj[f], obj); });
+ return obj;
+ };
+
+ // Memoize an expensive function by storing its results.
+ _.memoize = function(func, hasher) {
+ var memo = {};
+ hasher || (hasher = _.identity);
+ return function() {
+ var key = hasher.apply(this, arguments);
+ return _.has(memo, key) ? memo[key] : (memo[key] = func.apply(this, arguments));
+ };
+ };
+
+ // Delays a function for the given number of milliseconds, and then calls
+ // it with the arguments supplied.
+ _.delay = function(func, wait) {
+ var args = slice.call(arguments, 2);
+ return setTimeout(function(){ return func.apply(func, args); }, wait);
+ };
+
+ // Defers a function, scheduling it to run after the current call stack has
+ // cleared.
+ _.defer = function(func) {
+ return _.delay.apply(_, [func, 1].concat(slice.call(arguments, 1)));
+ };
+
+ // Returns a function, that, when invoked, will only be triggered at most once
+ // during a given window of time.
+ _.throttle = function(func, wait) {
+ var context, args, timeout, throttling, more;
+ var whenDone = _.debounce(function(){ more = throttling = false; }, wait);
+ return function() {
+ context = this; args = arguments;
+ var later = function() {
+ timeout = null;
+ if (more) func.apply(context, args);
+ whenDone();
+ };
+ if (!timeout) timeout = setTimeout(later, wait);
+ if (throttling) {
+ more = true;
+ } else {
+ func.apply(context, args);
+ }
+ whenDone();
+ throttling = true;
+ };
+ };
+
+ // Returns a function, that, as long as it continues to be invoked, will not
+ // be triggered. The function will be called after it stops being called for
+ // N milliseconds.
+ _.debounce = function(func, wait) {
+ var timeout;
+ return function() {
+ var context = this, args = arguments;
+ var later = function() {
+ timeout = null;
+ func.apply(context, args);
+ };
+ clearTimeout(timeout);
+ timeout = setTimeout(later, wait);
+ };
+ };
+
+ // Returns a function that will be executed at most one time, no matter how
+ // often you call it. Useful for lazy initialization.
+ _.once = function(func) {
+ var ran = false, memo;
+ return function() {
+ if (ran) return memo;
+ ran = true;
+ return memo = func.apply(this, arguments);
+ };
+ };
+
+ // Returns the first function passed as an argument to the second,
+ // allowing you to adjust arguments, run code before and after, and
+ // conditionally execute the original function.
+ _.wrap = function(func, wrapper) {
+ return function() {
+ var args = [func].concat(slice.call(arguments, 0));
+ return wrapper.apply(this, args);
+ };
+ };
+
+ // Returns a function that is the composition of a list of functions, each
+ // consuming the return value of the function that follows.
+ _.compose = function() {
+ var funcs = arguments;
+ return function() {
+ var args = arguments;
+ for (var i = funcs.length - 1; i >= 0; i--) {
+ args = [funcs[i].apply(this, args)];
+ }
+ return args[0];
+ };
+ };
+
+ // Returns a function that will only be executed after being called N times.
+ _.after = function(times, func) {
+ if (times <= 0) return func();
+ return function() {
+ if (--times < 1) { return func.apply(this, arguments); }
+ };
+ };
+
+ // Object Functions
+ // ----------------
+
+ // Retrieve the names of an object's properties.
+ // Delegates to **ECMAScript 5**'s native `Object.keys`
+ _.keys = nativeKeys || function(obj) {
+ if (obj !== Object(obj)) throw new TypeError('Invalid object');
+ var keys = [];
+ for (var key in obj) if (_.has(obj, key)) keys[keys.length] = key;
+ return keys;
+ };
+
+ // Retrieve the values of an object's properties.
+ _.values = function(obj) {
+ return _.map(obj, _.identity);
+ };
+
+ // Return a sorted list of the function names available on the object.
+ // Aliased as `methods`
+ _.functions = _.methods = function(obj) {
+ var names = [];
+ for (var key in obj) {
+ if (_.isFunction(obj[key])) names.push(key);
+ }
+ return names.sort();
+ };
+
+ // Extend a given object with all the properties in passed-in object(s).
+ _.extend = function(obj) {
+ each(slice.call(arguments, 1), function(source) {
+ for (var prop in source) {
+ obj[prop] = source[prop];
+ }
+ });
+ return obj;
+ };
+
+ // Fill in a given object with default properties.
+ _.defaults = function(obj) {
+ each(slice.call(arguments, 1), function(source) {
+ for (var prop in source) {
+ if (obj[prop] == null) obj[prop] = source[prop];
+ }
+ });
+ return obj;
+ };
+
+ // Create a (shallow-cloned) duplicate of an object.
+ _.clone = function(obj) {
+ if (!_.isObject(obj)) return obj;
+ return _.isArray(obj) ? obj.slice() : _.extend({}, obj);
+ };
+
+ // Invokes interceptor with the obj, and then returns obj.
+ // The primary purpose of this method is to "tap into" a method chain, in
+ // order to perform operations on intermediate results within the chain.
+ _.tap = function(obj, interceptor) {
+ interceptor(obj);
+ return obj;
+ };
+
+ // Internal recursive comparison function.
+ function eq(a, b, stack) {
+ // Identical objects are equal. `0 === -0`, but they aren't identical.
+ // See the Harmony `egal` proposal: http://wiki.ecmascript.org/doku.php?id=harmony:egal.
+ if (a === b) return a !== 0 || 1 / a == 1 / b;
+ // A strict comparison is necessary because `null == undefined`.
+ if (a == null || b == null) return a === b;
+ // Unwrap any wrapped objects.
+ if (a._chain) a = a._wrapped;
+ if (b._chain) b = b._wrapped;
+ // Invoke a custom `isEqual` method if one is provided.
+ if (a.isEqual && _.isFunction(a.isEqual)) return a.isEqual(b);
+ if (b.isEqual && _.isFunction(b.isEqual)) return b.isEqual(a);
+ // Compare `[[Class]]` names.
+ var className = toString.call(a);
+ if (className != toString.call(b)) return false;
+ switch (className) {
+ // Strings, numbers, dates, and booleans are compared by value.
+ case '[object String]':
+ // Primitives and their corresponding object wrappers are equivalent; thus, `"5"` is
+ // equivalent to `new String("5")`.
+ return a == String(b);
+ case '[object Number]':
+ // `NaN`s are equivalent, but non-reflexive. An `egal` comparison is performed for
+ // other numeric values.
+ return a != +a ? b != +b : (a == 0 ? 1 / a == 1 / b : a == +b);
+ case '[object Date]':
+ case '[object Boolean]':
+ // Coerce dates and booleans to numeric primitive values. Dates are compared by their
+ // millisecond representations. Note that invalid dates with millisecond representations
+ // of `NaN` are not equivalent.
+ return +a == +b;
+ // RegExps are compared by their source patterns and flags.
+ case '[object RegExp]':
+ return a.source == b.source &&
+ a.global == b.global &&
+ a.multiline == b.multiline &&
+ a.ignoreCase == b.ignoreCase;
+ }
+ if (typeof a != 'object' || typeof b != 'object') return false;
+ // Assume equality for cyclic structures. The algorithm for detecting cyclic
+ // structures is adapted from ES 5.1 section 15.12.3, abstract operation `JO`.
+ var length = stack.length;
+ while (length--) {
+ // Linear search. Performance is inversely proportional to the number of
+ // unique nested structures.
+ if (stack[length] == a) return true;
+ }
+ // Add the first object to the stack of traversed objects.
+ stack.push(a);
+ var size = 0, result = true;
+ // Recursively compare objects and arrays.
+ if (className == '[object Array]') {
+ // Compare array lengths to determine if a deep comparison is necessary.
+ size = a.length;
+ result = size == b.length;
+ if (result) {
+ // Deep compare the contents, ignoring non-numeric properties.
+ while (size--) {
+ // Ensure commutative equality for sparse arrays.
+ if (!(result = size in a == size in b && eq(a[size], b[size], stack))) break;
+ }
+ }
+ } else {
+ // Objects with different constructors are not equivalent.
+ if ('constructor' in a != 'constructor' in b || a.constructor != b.constructor) return false;
+ // Deep compare objects.
+ for (var key in a) {
+ if (_.has(a, key)) {
+ // Count the expected number of properties.
+ size++;
+ // Deep compare each member.
+ if (!(result = _.has(b, key) && eq(a[key], b[key], stack))) break;
+ }
+ }
+ // Ensure that both objects contain the same number of properties.
+ if (result) {
+ for (key in b) {
+ if (_.has(b, key) && !(size--)) break;
+ }
+ result = !size;
+ }
+ }
+ // Remove the first object from the stack of traversed objects.
+ stack.pop();
+ return result;
+ }
+
+ // Perform a deep comparison to check if two objects are equal.
+ _.isEqual = function(a, b) {
+ return eq(a, b, []);
+ };
+
+ // Is a given array, string, or object empty?
+ // An "empty" object has no enumerable own-properties.
+ _.isEmpty = function(obj) {
+ if (_.isArray(obj) || _.isString(obj)) return obj.length === 0;
+ for (var key in obj) if (_.has(obj, key)) return false;
+ return true;
+ };
+
+ // Is a given value a DOM element?
+ _.isElement = function(obj) {
+ return !!(obj && obj.nodeType == 1);
+ };
+
+ // Is a given value an array?
+ // Delegates to ECMA5's native Array.isArray
+ _.isArray = nativeIsArray || function(obj) {
+ return toString.call(obj) == '[object Array]';
+ };
+
+ // Is a given variable an object?
+ _.isObject = function(obj) {
+ return obj === Object(obj);
+ };
+
+ // Is a given variable an arguments object?
+ _.isArguments = function(obj) {
+ return toString.call(obj) == '[object Arguments]';
+ };
+ if (!_.isArguments(arguments)) {
+ _.isArguments = function(obj) {
+ return !!(obj && _.has(obj, 'callee'));
+ };
+ }
+
+ // Is a given value a function?
+ _.isFunction = function(obj) {
+ return toString.call(obj) == '[object Function]';
+ };
+
+ // Is a given value a string?
+ _.isString = function(obj) {
+ return toString.call(obj) == '[object String]';
+ };
+
+ // Is a given value a number?
+ _.isNumber = function(obj) {
+ return toString.call(obj) == '[object Number]';
+ };
+
+ // Is the given value `NaN`?
+ _.isNaN = function(obj) {
+ // `NaN` is the only value for which `===` is not reflexive.
+ return obj !== obj;
+ };
+
+ // Is a given value a boolean?
+ _.isBoolean = function(obj) {
+ return obj === true || obj === false || toString.call(obj) == '[object Boolean]';
+ };
+
+ // Is a given value a date?
+ _.isDate = function(obj) {
+ return toString.call(obj) == '[object Date]';
+ };
+
+ // Is the given value a regular expression?
+ _.isRegExp = function(obj) {
+ return toString.call(obj) == '[object RegExp]';
+ };
+
+ // Is a given value equal to null?
+ _.isNull = function(obj) {
+ return obj === null;
+ };
+
+ // Is a given variable undefined?
+ _.isUndefined = function(obj) {
+ return obj === void 0;
+ };
+
+ // Has own property?
+ _.has = function(obj, key) {
+ return hasOwnProperty.call(obj, key);
+ };
+
+ // Utility Functions
+ // -----------------
+
+ // Run Underscore.js in *noConflict* mode, returning the `_` variable to its
+ // previous owner. Returns a reference to the Underscore object.
+ _.noConflict = function() {
+ root._ = previousUnderscore;
+ return this;
+ };
+
+ // Keep the identity function around for default iterators.
+ _.identity = function(value) {
+ return value;
+ };
+
+ // Run a function **n** times.
+ _.times = function (n, iterator, context) {
+ for (var i = 0; i < n; i++) iterator.call(context, i);
+ };
+
+ // Escape a string for HTML interpolation.
+ _.escape = function(string) {
+ return (''+string).replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>').replace(/"/g, '"').replace(/'/g, ''').replace(/\//g,'/');
+ };
+
+ // Add your own custom functions to the Underscore object, ensuring that
+ // they're correctly added to the OOP wrapper as well.
+ _.mixin = function(obj) {
+ each(_.functions(obj), function(name){
+ addToWrapper(name, _[name] = obj[name]);
+ });
+ };
+
+ // Generate a unique integer id (unique within the entire client session).
+ // Useful for temporary DOM ids.
+ var idCounter = 0;
+ _.uniqueId = function(prefix) {
+ var id = idCounter++;
+ return prefix ? prefix + id : id;
+ };
+
+ // By default, Underscore uses ERB-style template delimiters, change the
+ // following template settings to use alternative delimiters.
+ _.templateSettings = {
+ evaluate : /<%([\s\S]+?)%>/g,
+ interpolate : /<%=([\s\S]+?)%>/g,
+ escape : /<%-([\s\S]+?)%>/g
+ };
+
+ // When customizing `templateSettings`, if you don't want to define an
+ // interpolation, evaluation or escaping regex, we need one that is
+ // guaranteed not to match.
+ var noMatch = /.^/;
+
+ // Within an interpolation, evaluation, or escaping, remove HTML escaping
+ // that had been previously added.
+ var unescape = function(code) {
+ return code.replace(/\\\\/g, '\\').replace(/\\'/g, "'");
+ };
+
+ // JavaScript micro-templating, similar to John Resig's implementation.
+ // Underscore templating handles arbitrary delimiters, preserves whitespace,
+ // and correctly escapes quotes within interpolated code.
+ _.template = function(str, data) {
+ var c = _.templateSettings;
+ var tmpl = 'var __p=[],print=function(){__p.push.apply(__p,arguments);};' +
+ 'with(obj||{}){__p.push(\'' +
+ str.replace(/\\/g, '\\\\')
+ .replace(/'/g, "\\'")
+ .replace(c.escape || noMatch, function(match, code) {
+ return "',_.escape(" + unescape(code) + "),'";
+ })
+ .replace(c.interpolate || noMatch, function(match, code) {
+ return "'," + unescape(code) + ",'";
+ })
+ .replace(c.evaluate || noMatch, function(match, code) {
+ return "');" + unescape(code).replace(/[\r\n\t]/g, ' ') + ";__p.push('";
+ })
+ .replace(/\r/g, '\\r')
+ .replace(/\n/g, '\\n')
+ .replace(/\t/g, '\\t')
+ + "');}return __p.join('');";
+ var func = new Function('obj', '_', tmpl);
+ if (data) return func(data, _);
+ return function(data) {
+ return func.call(this, data, _);
+ };
+ };
+
+ // Add a "chain" function, which will delegate to the wrapper.
+ _.chain = function(obj) {
+ return _(obj).chain();
+ };
+
+ // The OOP Wrapper
+ // ---------------
+
+ // If Underscore is called as a function, it returns a wrapped object that
+ // can be used OO-style. This wrapper holds altered versions of all the
+ // underscore functions. Wrapped objects may be chained.
+ var wrapper = function(obj) { this._wrapped = obj; };
+
+ // Expose `wrapper.prototype` as `_.prototype`
+ _.prototype = wrapper.prototype;
+
+ // Helper function to continue chaining intermediate results.
+ var result = function(obj, chain) {
+ return chain ? _(obj).chain() : obj;
+ };
+
+ // A method to easily add functions to the OOP wrapper.
+ var addToWrapper = function(name, func) {
+ wrapper.prototype[name] = function() {
+ var args = slice.call(arguments);
+ unshift.call(args, this._wrapped);
+ return result(func.apply(_, args), this._chain);
+ };
+ };
+
+ // Add all of the Underscore functions to the wrapper object.
+ _.mixin(_);
+
+ // Add all mutator Array functions to the wrapper.
+ each(['pop', 'push', 'reverse', 'shift', 'sort', 'splice', 'unshift'], function(name) {
+ var method = ArrayProto[name];
+ wrapper.prototype[name] = function() {
+ var wrapped = this._wrapped;
+ method.apply(wrapped, arguments);
+ var length = wrapped.length;
+ if ((name == 'shift' || name == 'splice') && length === 0) delete wrapped[0];
+ return result(wrapped, this._chain);
+ };
+ });
+
+ // Add all accessor Array functions to the wrapper.
+ each(['concat', 'join', 'slice'], function(name) {
+ var method = ArrayProto[name];
+ wrapper.prototype[name] = function() {
+ return result(method.apply(this._wrapped, arguments), this._chain);
+ };
+ });
+
+ // Start chaining a wrapped Underscore object.
+ wrapper.prototype.chain = function() {
+ this._chain = true;
+ return this;
+ };
+
+ // Extracts the result from a wrapped and chained object.
+ wrapper.prototype.value = function() {
+ return this._wrapped;
+ };
+
+}).call(this);
[28/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/bash_operator.html
----------------------------------------------------------------------
diff --git a/_modules/bash_operator.html b/_modules/bash_operator.html
new file mode 100644
index 0000000..98de858
--- /dev/null
+++ b/_modules/bash_operator.html
@@ -0,0 +1,290 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>bash_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>bash_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for bash_operator</h1><div class="highlight"><pre>
+<span></span>
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">bytes</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">from</span> <span class="nn">subprocess</span> <span class="kn">import</span> <span class="n">Popen</span><span class="p">,</span> <span class="n">STDOUT</span><span class="p">,</span> <span class="n">PIPE</span>
+<span class="kn">from</span> <span class="nn">tempfile</span> <span class="kn">import</span> <span class="n">gettempdir</span><span class="p">,</span> <span class="n">NamedTemporaryFile</span>
+
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.file</span> <span class="kn">import</span> <span class="n">TemporaryDirectory</span>
+
+
+<div class="viewcode-block" id="BashOperator"><a class="viewcode-back" href="../code.html#airflow.operators.BashOperator">[docs]</a><span class="k">class</span> <span class="nc">BashOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Execute a Bash script, command or set of commands.</span>
+
+<span class="sd"> :param bash_command: The command, set of commands or reference to a</span>
+<span class="sd"> bash script (must be '.sh') to be executed.</span>
+<span class="sd"> :type bash_command: string</span>
+<span class="sd"> :param env: If env is not None, it must be a mapping that defines the</span>
+<span class="sd"> environment variables for the new process; these are used instead</span>
+<span class="sd"> of inheriting the current process environment, which is the default</span>
+<span class="sd"> behavior. (templated)</span>
+<span class="sd"> :type env: dict</span>
+<span class="sd"> :type output_encoding: output encoding of bash command</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'bash_command'</span><span class="p">,</span> <span class="s1">'env'</span><span class="p">)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sh'</span><span class="p">,</span> <span class="s1">'.bash'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#f0ede4'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">bash_command</span><span class="p">,</span>
+ <span class="n">xcom_push</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">env</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">output_encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> If xcom_push is True, the last line written to stdout will also</span>
+<span class="sd"> be pushed to an XCom when the bash command completes.</span>
+<span class="sd"> """</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">BashOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bash_command</span> <span class="o">=</span> <span class="n">bash_command</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">env</span> <span class="o">=</span> <span class="n">env</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">xcom_push_flag</span> <span class="o">=</span> <span class="n">xcom_push</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">output_encoding</span> <span class="o">=</span> <span class="n">output_encoding</span>
+
+<div class="viewcode-block" id="BashOperator.execute"><a class="viewcode-back" href="../code.html#airflow.operators.BashOperator.execute">[docs]</a> <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Execute the bash command in a temporary directory</span>
+<span class="sd"> which will be cleaned afterwards</span>
+<span class="sd"> """</span>
+ <span class="n">bash_command</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">bash_command</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"tmp dir root location: </span><span class="se">\n</span><span class="s2">"</span> <span class="o">+</span> <span class="n">gettempdir</span><span class="p">())</span>
+ <span class="k">with</span> <span class="n">TemporaryDirectory</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="s1">'airflowtmp'</span><span class="p">)</span> <span class="k">as</span> <span class="n">tmp_dir</span><span class="p">:</span>
+ <span class="k">with</span> <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="nb">dir</span><span class="o">=</span><span class="n">tmp_dir</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">task_id</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+
+ <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="nb">bytes</span><span class="p">(</span><span class="n">bash_command</span><span class="p">,</span> <span class="s1">'utf_8'</span><span class="p">))</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
+ <span class="n">fname</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span>
+ <span class="n">script_location</span> <span class="o">=</span> <span class="n">tmp_dir</span> <span class="o">+</span> <span class="s2">"/"</span> <span class="o">+</span> <span class="n">fname</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Temporary script "</span>
+ <span class="s2">"location :{0}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">script_location</span><span class="p">))</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running command: "</span> <span class="o">+</span> <span class="n">bash_command</span><span class="p">)</span>
+ <span class="n">sp</span> <span class="o">=</span> <span class="n">Popen</span><span class="p">(</span>
+ <span class="p">[</span><span class="s1">'bash'</span><span class="p">,</span> <span class="n">fname</span><span class="p">],</span>
+ <span class="n">stdout</span><span class="o">=</span><span class="n">PIPE</span><span class="p">,</span> <span class="n">stderr</span><span class="o">=</span><span class="n">STDOUT</span><span class="p">,</span>
+ <span class="n">cwd</span><span class="o">=</span><span class="n">tmp_dir</span><span class="p">,</span> <span class="n">env</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">env</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">sp</span> <span class="o">=</span> <span class="n">sp</span>
+
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Output:"</span><span class="p">)</span>
+ <span class="n">line</span> <span class="o">=</span> <span class="s1">''</span>
+ <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="nb">iter</span><span class="p">(</span><span class="n">sp</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">readline</span><span class="p">,</span> <span class="n">b</span><span class="s1">''</span><span class="p">):</span>
+ <span class="n">line</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">output_encoding</span><span class="p">)</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
+ <span class="n">sp</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Command exited with "</span>
+ <span class="s2">"return code {0}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">sp</span><span class="o">.</span><span class="n">returncode</span><span class="p">))</span>
+
+ <span class="k">if</span> <span class="n">sp</span><span class="o">.</span><span class="n">returncode</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Bash command failed"</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">xcom_push_flag</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">line</span></div>
+
+ <span class="k">def</span> <span class="nf">on_kill</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Sending SIGTERM signal to bash subprocess'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sp</span><span class="o">.</span><span class="n">terminate</span><span class="p">()</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/cloudant_hook.html
----------------------------------------------------------------------
diff --git a/_modules/cloudant_hook.html b/_modules/cloudant_hook.html
new file mode 100644
index 0000000..4c5bb1f
--- /dev/null
+++ b/_modules/cloudant_hook.html
@@ -0,0 +1,274 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>cloudant_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>cloudant_hook</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for cloudant_hook</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">import</span> <span class="nn">cloudant</span>
+
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="kn">import</span> <span class="n">BaseHook</span>
+
+
+<div class="viewcode-block" id="CloudantHook"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.CloudantHook">[docs]</a><span class="k">class</span> <span class="nc">CloudantHook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
+ <span class="sd">"""Interact with Cloudant.</span>
+
+<span class="sd"> This class is a thin wrapper around the cloudant python library. See the</span>
+<span class="sd"> documentation `here <https://github.com/cloudant-labs/cloudant-python>`_.</span>
+<span class="sd"> """</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">cloudant_conn_id</span><span class="o">=</span><span class="s1">'cloudant_default'</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">CloudantHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="s1">'cloudant'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cloudant_conn_id</span> <span class="o">=</span> <span class="n">cloudant_conn_id</span>
+
+ <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">_str</span><span class="p">(</span><span class="n">s</span><span class="p">):</span>
+ <span class="c1"># cloudant-python doesn't support unicode.</span>
+ <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="nb">unicode</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">debug</span><span class="p">((</span><span class="s1">'cloudant-python does not support unicode. '</span>
+ <span class="s1">'Encoding </span><span class="si">%s</span><span class="s1"> as ascii using "ignore".'</span><span class="p">),</span>
+ <span class="n">s</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">s</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'ascii'</span><span class="p">,</span> <span class="s1">'ignore'</span><span class="p">)</span>
+
+ <span class="k">return</span> <span class="n">s</span>
+
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">cloudant_conn_id</span><span class="p">)</span>
+
+ <span class="k">for</span> <span class="n">conn_param</span> <span class="ow">in</span> <span class="p">[</span><span class="s1">'host'</span><span class="p">,</span> <span class="s1">'password'</span><span class="p">,</span> <span class="s1">'schema'</span><span class="p">]:</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">conn_param</span><span class="p">)</span> <span class="ow">or</span> <span class="ow">not</span> <span class="nb">getattr</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">conn_param</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span>
+ <span class="s1">'missing connection parameter {0}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">conn_param</span><span class="p">)</span>
+ <span class="p">)</span>
+
+ <span class="c1"># In the connection form:</span>
+ <span class="c1"># - 'host' is renamed to 'Account'</span>
+ <span class="c1"># - 'login' is renamed 'Username (or API Key)'</span>
+ <span class="c1"># - 'schema' is renamed to 'Database'</span>
+ <span class="c1">#</span>
+ <span class="c1"># So, use the 'host' attribute as the account name, and, if login is</span>
+ <span class="c1"># defined, use that as the username.</span>
+ <span class="n">account</span> <span class="o">=</span> <span class="n">cloudant</span><span class="o">.</span><span class="n">Account</span><span class="p">(</span><span class="n">_str</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">host</span><span class="p">))</span>
+
+ <span class="n">username</span> <span class="o">=</span> <span class="n">_str</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">login</span> <span class="ow">or</span> <span class="n">conn</span><span class="o">.</span><span class="n">host</span><span class="p">)</span>
+
+ <span class="n">account</span><span class="o">.</span><span class="n">login</span><span class="p">(</span>
+ <span class="n">username</span><span class="p">,</span>
+ <span class="n">_str</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">password</span><span class="p">))</span><span class="o">.</span><span class="n">raise_for_status</span><span class="p">()</span>
+
+ <span class="k">return</span> <span class="n">account</span><span class="o">.</span><span class="n">database</span><span class="p">(</span><span class="n">_str</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">schema</span><span class="p">))</span>
+
+<div class="viewcode-block" id="CloudantHook.db"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.CloudantHook.db">[docs]</a> <span class="k">def</span> <span class="nf">db</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""Returns the Database object for this hook.</span>
+
+<span class="sd"> See the documentation for cloudant-python here</span>
+<span class="sd"> https://github.com/cloudant-labs/cloudant-python.</span>
+<span class="sd"> """</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span></div></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/dagrun_operator.html
----------------------------------------------------------------------
diff --git a/_modules/dagrun_operator.html b/_modules/dagrun_operator.html
new file mode 100644
index 0000000..66f5cef
--- /dev/null
+++ b/_modules/dagrun_operator.html
@@ -0,0 +1,260 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>dagrun_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>dagrun_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for dagrun_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span><span class="p">,</span> <span class="n">DagRun</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="kn">import</span> <span class="n">settings</span>
+
+
+<span class="k">class</span> <span class="nc">DagRunOrder</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">run_id</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">payload</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">run_id</span> <span class="o">=</span> <span class="n">run_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">payload</span> <span class="o">=</span> <span class="n">payload</span>
+
+
+<div class="viewcode-block" id="TriggerDagRunOperator"><a class="viewcode-back" href="../code.html#airflow.operators.TriggerDagRunOperator">[docs]</a><span class="k">class</span> <span class="nc">TriggerDagRunOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Triggers a DAG run for a specified ``dag_id`` if a criteria is met</span>
+
+<span class="sd"> :param trigger_dag_id: the dag_id to trigger</span>
+<span class="sd"> :type trigger_dag_id: str</span>
+<span class="sd"> :param python_callable: a reference to a python function that will be</span>
+<span class="sd"> called while passing it the ``context`` object and a placeholder</span>
+<span class="sd"> object ``obj`` for your callable to fill and return if you want</span>
+<span class="sd"> a DagRun created. This ``obj`` object contains a ``run_id`` and</span>
+<span class="sd"> ``payload`` attribute that you can modify in your function.</span>
+<span class="sd"> The ``run_id`` should be a unique identifier for that DAG run, and</span>
+<span class="sd"> the payload has to be a picklable object that will be made available</span>
+<span class="sd"> to your tasks while executing that DAG run. Your function header</span>
+<span class="sd"> should look like ``def foo(context, dag_run_obj):``</span>
+<span class="sd"> :type python_callable: python callable</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">()</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">()</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#ffefeb'</span>
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">trigger_dag_id</span><span class="p">,</span>
+ <span class="n">python_callable</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">TriggerDagRunOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">python_callable</span> <span class="o">=</span> <span class="n">python_callable</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">trigger_dag_id</span> <span class="o">=</span> <span class="n">trigger_dag_id</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">dro</span> <span class="o">=</span> <span class="n">DagRunOrder</span><span class="p">(</span><span class="n">run_id</span><span class="o">=</span><span class="s1">'trig__'</span> <span class="o">+</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span><span class="o">.</span><span class="n">isoformat</span><span class="p">())</span>
+ <span class="n">dro</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">python_callable</span><span class="p">(</span><span class="n">context</span><span class="p">,</span> <span class="n">dro</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">dro</span><span class="p">:</span>
+ <span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
+ <span class="n">dr</span> <span class="o">=</span> <span class="n">DagRun</span><span class="p">(</span>
+ <span class="n">dag_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">trigger_dag_id</span><span class="p">,</span>
+ <span class="n">run_id</span><span class="o">=</span><span class="n">dro</span><span class="o">.</span><span class="n">run_id</span><span class="p">,</span>
+ <span class="n">conf</span><span class="o">=</span><span class="n">dro</span><span class="o">.</span><span class="n">payload</span><span class="p">,</span>
+ <span class="n">external_trigger</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Creating DagRun {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">dr</span><span class="p">))</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">dr</span><span class="p">)</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Criteria not met, moving on"</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/dbapi_hook.html
----------------------------------------------------------------------
diff --git a/_modules/dbapi_hook.html b/_modules/dbapi_hook.html
new file mode 100644
index 0000000..06bbb28
--- /dev/null
+++ b/_modules/dbapi_hook.html
@@ -0,0 +1,426 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>dbapi_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>dbapi_hook</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for dbapi_hook</h1><div class="highlight"><pre>
+<span></span>
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">str</span>
+<span class="kn">from</span> <span class="nn">past.builtins</span> <span class="kn">import</span> <span class="nb">basestring</span>
+<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
+<span class="kn">import</span> <span class="nn">numpy</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="kn">import</span> <span class="n">BaseHook</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+
+
+<div class="viewcode-block" id="DbApiHook"><a class="viewcode-back" href="../code.html#airflow.hooks.DbApiHook">[docs]</a><span class="k">class</span> <span class="nc">DbApiHook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Abstract base class for sql hooks.</span>
+<span class="sd"> """</span>
+ <span class="c1"># Override to provide the connection name.</span>
+ <span class="n">conn_name_attr</span> <span class="o">=</span> <span class="bp">None</span>
+ <span class="c1"># Override to have a default connection id for a particular dbHook</span>
+ <span class="n">default_conn_name</span> <span class="o">=</span> <span class="s1">'default_conn_id'</span>
+ <span class="c1"># Override if this db supports autocommit.</span>
+ <span class="n">supports_autocommit</span> <span class="o">=</span> <span class="bp">False</span>
+ <span class="c1"># Override with the object that exposes the connect method</span>
+ <span class="n">connector</span> <span class="o">=</span> <span class="bp">None</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn_name_attr</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"conn_name_attr is not defined"</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
+ <span class="nb">setattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn_name_attr</span><span class="p">,</span> <span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
+ <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn_name_attr</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">kwargs</span><span class="p">:</span>
+ <span class="nb">setattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn_name_attr</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">default_conn_name</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="nb">setattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn_name_attr</span><span class="p">,</span> <span class="n">kwargs</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">conn_name_attr</span><span class="p">])</span>
+
+<div class="viewcode-block" id="DbApiHook.get_conn"><a class="viewcode-back" href="../code.html#airflow.hooks.DbApiHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""Returns a connection object</span>
+<span class="sd"> """</span>
+ <span class="n">db</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="nb">getattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn_name_attr</span><span class="p">))</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">connector</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span>
+ <span class="n">host</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">host</span><span class="p">,</span>
+ <span class="n">port</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">port</span><span class="p">,</span>
+ <span class="n">username</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">login</span><span class="p">,</span>
+ <span class="n">schema</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">schema</span><span class="p">)</span></div>
+
+
+<div class="viewcode-block" id="DbApiHook.get_pandas_df"><a class="viewcode-back" href="../code.html#airflow.hooks.DbApiHook.get_pandas_df">[docs]</a> <span class="k">def</span> <span class="nf">get_pandas_df</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> Executes the sql and returns a pandas dataframe</span>
+
+<span class="sd"> :param sql: the sql statement to be executed (str) or a list of</span>
+<span class="sd"> sql statements to execute</span>
+<span class="sd"> :type sql: str or list</span>
+<span class="sd"> :param parameters: The parameters to render the SQL query with.</span>
+<span class="sd"> :type parameters: mapping or iterable</span>
+<span class="sd"> '''</span>
+ <span class="kn">import</span> <span class="nn">pandas.io.sql</span> <span class="kn">as</span> <span class="nn">psql</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">df</span> <span class="o">=</span> <span class="n">psql</span><span class="o">.</span><span class="n">read_sql</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="n">con</span><span class="o">=</span><span class="n">conn</span><span class="p">,</span> <span class="n">params</span><span class="o">=</span><span class="n">parameters</span><span class="p">)</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">df</span></div>
+
+<div class="viewcode-block" id="DbApiHook.get_records"><a class="viewcode-back" href="../code.html#airflow.hooks.DbApiHook.get_records">[docs]</a> <span class="k">def</span> <span class="nf">get_records</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> Executes the sql and returns a set of records.</span>
+
+<span class="sd"> :param sql: the sql statement to be executed (str) or a list of</span>
+<span class="sd"> sql statements to execute</span>
+<span class="sd"> :type sql: str or list</span>
+<span class="sd"> :param parameters: The parameters to render the SQL query with.</span>
+<span class="sd"> :type parameters: mapping or iterable</span>
+<span class="sd"> '''</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">cur</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_cursor</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">parameters</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="n">parameters</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">)</span>
+ <span class="n">rows</span> <span class="o">=</span> <span class="n">cur</span><span class="o">.</span><span class="n">fetchall</span><span class="p">()</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">rows</span></div>
+
+<div class="viewcode-block" id="DbApiHook.get_first"><a class="viewcode-back" href="../code.html#airflow.hooks.DbApiHook.get_first">[docs]</a> <span class="k">def</span> <span class="nf">get_first</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> Executes the sql and returns the first resulting row.</span>
+
+<span class="sd"> :param sql: the sql statement to be executed (str) or a list of</span>
+<span class="sd"> sql statements to execute</span>
+<span class="sd"> :type sql: str or list</span>
+<span class="sd"> :param parameters: The parameters to render the SQL query with.</span>
+<span class="sd"> :type parameters: mapping or iterable</span>
+<span class="sd"> '''</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">cur</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">parameters</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="n">parameters</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">)</span>
+ <span class="n">rows</span> <span class="o">=</span> <span class="n">cur</span><span class="o">.</span><span class="n">fetchone</span><span class="p">()</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">rows</span></div>
+
+<div class="viewcode-block" id="DbApiHook.run"><a class="viewcode-back" href="../code.html#airflow.hooks.DbApiHook.run">[docs]</a> <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="n">autocommit</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Runs a command or a list of commands. Pass a list of sql</span>
+<span class="sd"> statements to the sql parameter to get them to execute</span>
+<span class="sd"> sequentially</span>
+
+<span class="sd"> :param sql: the sql statement to be executed (str) or a list of</span>
+<span class="sd"> sql statements to execute</span>
+<span class="sd"> :type sql: str or list</span>
+<span class="sd"> :param autocommit: What to set the connection's autocommit setting to</span>
+<span class="sd"> before executing the query.</span>
+<span class="sd"> :type autocommit: bool</span>
+<span class="sd"> :param parameters: The parameters to render the SQL query with.</span>
+<span class="sd"> :type parameters: mapping or iterable</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="nb">basestring</span><span class="p">):</span>
+ <span class="n">sql</span> <span class="o">=</span> <span class="p">[</span><span class="n">sql</span><span class="p">]</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">supports_autocommit</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">set_autocommit</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">autocommit</span><span class="p">)</span>
+
+ <span class="n">cur</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
+ <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">sql</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">parameters</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">parameters</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span></div>
+
+ <span class="k">def</span> <span class="nf">set_autocommit</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">conn</span><span class="p">,</span> <span class="n">autocommit</span><span class="p">):</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">autocommit</span> <span class="o">=</span> <span class="n">autocommit</span>
+
+<div class="viewcode-block" id="DbApiHook.get_cursor"><a class="viewcode-back" href="../code.html#airflow.hooks.DbApiHook.get_cursor">[docs]</a> <span class="k">def</span> <span class="nf">get_cursor</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a cursor</span>
+<span class="sd"> """</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span></div>
+
+<div class="viewcode-block" id="DbApiHook.insert_rows"><a class="viewcode-back" href="../code.html#airflow.hooks.DbApiHook.insert_rows">[docs]</a> <span class="k">def</span> <span class="nf">insert_rows</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">rows</span><span class="p">,</span> <span class="n">target_fields</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">commit_every</span><span class="o">=</span><span class="mi">1000</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> A generic way to insert a set of tuples into a table,</span>
+<span class="sd"> the whole set of inserts is treated as one transaction</span>
+
+<span class="sd"> :param table: Name of the target table</span>
+<span class="sd"> :type table: str</span>
+<span class="sd"> :param rows: The rows to insert into the table</span>
+<span class="sd"> :type rows: iterable of tuples</span>
+<span class="sd"> :param target_fields: The names of the columns to fill in the table</span>
+<span class="sd"> :type target_fields: iterable of strings</span>
+<span class="sd"> :param commit_every: The maximum number of rows to insert in one</span>
+<span class="sd"> transaction. Set to 0 to insert all rows in one transaction.</span>
+<span class="sd"> :type commit_every: int</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="n">target_fields</span><span class="p">:</span>
+ <span class="n">target_fields</span> <span class="o">=</span> <span class="s2">", "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">target_fields</span><span class="p">)</span>
+ <span class="n">target_fields</span> <span class="o">=</span> <span class="s2">"({})"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">target_fields</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">target_fields</span> <span class="o">=</span> <span class="s1">''</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">cur</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">supports_autocommit</span><span class="p">:</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s1">'SET autocommit = 0'</span><span class="p">)</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
+ <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">rows</span><span class="p">:</span>
+ <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
+ <span class="n">l</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="k">for</span> <span class="n">cell</span> <span class="ow">in</span> <span class="n">row</span><span class="p">:</span>
+ <span class="n">l</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_serialize_cell</span><span class="p">(</span><span class="n">cell</span><span class="p">))</span>
+ <span class="n">values</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">(</span><span class="n">l</span><span class="p">)</span>
+ <span class="n">sql</span> <span class="o">=</span> <span class="s2">"INSERT INTO {0} {1} VALUES ({2});"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">table</span><span class="p">,</span>
+ <span class="n">target_fields</span><span class="p">,</span>
+ <span class="s2">","</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">values</span><span class="p">))</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">commit_every</span> <span class="ow">and</span> <span class="n">i</span> <span class="o">%</span> <span class="n">commit_every</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s2">"Loaded {i} into {table} rows so far"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s2">"Done loading. Loaded a total of {i} rows"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span></div>
+
+ <span class="nd">@staticmethod</span>
+ <span class="k">def</span> <span class="nf">_serialize_cell</span><span class="p">(</span><span class="n">cell</span><span class="p">):</span>
+ <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">cell</span><span class="p">,</span> <span class="nb">basestring</span><span class="p">):</span>
+ <span class="k">return</span> <span class="s2">"'"</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">cell</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">"'"</span><span class="p">,</span> <span class="s2">"''"</span><span class="p">)</span> <span class="o">+</span> <span class="s2">"'"</span>
+ <span class="k">elif</span> <span class="n">cell</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
+ <span class="k">return</span> <span class="s1">'NULL'</span>
+ <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">cell</span><span class="p">,</span> <span class="n">numpy</span><span class="o">.</span><span class="n">datetime64</span><span class="p">):</span>
+ <span class="k">return</span> <span class="s2">"'"</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">cell</span><span class="p">)</span> <span class="o">+</span> <span class="s2">"'"</span>
+ <span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">cell</span><span class="p">,</span> <span class="n">datetime</span><span class="p">):</span>
+ <span class="k">return</span> <span class="s2">"'"</span> <span class="o">+</span> <span class="n">cell</span><span class="o">.</span><span class="n">isoformat</span><span class="p">()</span> <span class="o">+</span> <span class="s2">"'"</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="n">cell</span><span class="p">)</span>
+
+<div class="viewcode-block" id="DbApiHook.bulk_dump"><a class="viewcode-back" href="../code.html#airflow.hooks.DbApiHook.bulk_dump">[docs]</a> <span class="k">def</span> <span class="nf">bulk_dump</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">tmp_file</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Dumps a database table into a tab-delimited file</span>
+
+<span class="sd"> :param table: The name of the source table</span>
+<span class="sd"> :type table: str</span>
+<span class="sd"> :param tmp_file: The path of the target file</span>
+<span class="sd"> :type tmp_file: str</span>
+<span class="sd"> """</span>
+ <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">()</span></div>
+
+<div class="viewcode-block" id="DbApiHook.bulk_load"><a class="viewcode-back" href="../code.html#airflow.hooks.DbApiHook.bulk_load">[docs]</a> <span class="k">def</span> <span class="nf">bulk_load</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">tmp_file</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Loads a tab-delimited file into a database table</span>
+
+<span class="sd"> :param table: The name of the target table</span>
+<span class="sd"> :type table: str</span>
+<span class="sd"> :param tmp_file: The path of the file to load into the table</span>
+<span class="sd"> :type tmp_file: str</span>
+<span class="sd"> """</span>
+ <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">()</span></div></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[14/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/doctools.js
----------------------------------------------------------------------
diff --git a/_static/doctools.js b/_static/doctools.js
new file mode 100644
index 0000000..8163495
--- /dev/null
+++ b/_static/doctools.js
@@ -0,0 +1,287 @@
+/*
+ * doctools.js
+ * ~~~~~~~~~~~
+ *
+ * Sphinx JavaScript utilities for all documentation.
+ *
+ * :copyright: Copyright 2007-2016 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+/**
+ * select a different prefix for underscore
+ */
+$u = _.noConflict();
+
+/**
+ * make the code below compatible with browsers without
+ * an installed firebug like debugger
+if (!window.console || !console.firebug) {
+ var names = ["log", "debug", "info", "warn", "error", "assert", "dir",
+ "dirxml", "group", "groupEnd", "time", "timeEnd", "count", "trace",
+ "profile", "profileEnd"];
+ window.console = {};
+ for (var i = 0; i < names.length; ++i)
+ window.console[names[i]] = function() {};
+}
+ */
+
+/**
+ * small helper function to urldecode strings
+ */
+jQuery.urldecode = function(x) {
+ return decodeURIComponent(x).replace(/\+/g, ' ');
+};
+
+/**
+ * small helper function to urlencode strings
+ */
+jQuery.urlencode = encodeURIComponent;
+
+/**
+ * This function returns the parsed url parameters of the
+ * current request. Multiple values per key are supported,
+ * it will always return arrays of strings for the value parts.
+ */
+jQuery.getQueryParameters = function(s) {
+ if (typeof s == 'undefined')
+ s = document.location.search;
+ var parts = s.substr(s.indexOf('?') + 1).split('&');
+ var result = {};
+ for (var i = 0; i < parts.length; i++) {
+ var tmp = parts[i].split('=', 2);
+ var key = jQuery.urldecode(tmp[0]);
+ var value = jQuery.urldecode(tmp[1]);
+ if (key in result)
+ result[key].push(value);
+ else
+ result[key] = [value];
+ }
+ return result;
+};
+
+/**
+ * highlight a given string on a jquery object by wrapping it in
+ * span elements with the given class name.
+ */
+jQuery.fn.highlightText = function(text, className) {
+ function highlight(node) {
+ if (node.nodeType == 3) {
+ var val = node.nodeValue;
+ var pos = val.toLowerCase().indexOf(text);
+ if (pos >= 0 && !jQuery(node.parentNode).hasClass(className)) {
+ var span = document.createElement("span");
+ span.className = className;
+ span.appendChild(document.createTextNode(val.substr(pos, text.length)));
+ node.parentNode.insertBefore(span, node.parentNode.insertBefore(
+ document.createTextNode(val.substr(pos + text.length)),
+ node.nextSibling));
+ node.nodeValue = val.substr(0, pos);
+ }
+ }
+ else if (!jQuery(node).is("button, select, textarea")) {
+ jQuery.each(node.childNodes, function() {
+ highlight(this);
+ });
+ }
+ }
+ return this.each(function() {
+ highlight(this);
+ });
+};
+
+/*
+ * backward compatibility for jQuery.browser
+ * This will be supported until firefox bug is fixed.
+ */
+if (!jQuery.browser) {
+ jQuery.uaMatch = function(ua) {
+ ua = ua.toLowerCase();
+
+ var match = /(chrome)[ \/]([\w.]+)/.exec(ua) ||
+ /(webkit)[ \/]([\w.]+)/.exec(ua) ||
+ /(opera)(?:.*version|)[ \/]([\w.]+)/.exec(ua) ||
+ /(msie) ([\w.]+)/.exec(ua) ||
+ ua.indexOf("compatible") < 0 && /(mozilla)(?:.*? rv:([\w.]+)|)/.exec(ua) ||
+ [];
+
+ return {
+ browser: match[ 1 ] || "",
+ version: match[ 2 ] || "0"
+ };
+ };
+ jQuery.browser = {};
+ jQuery.browser[jQuery.uaMatch(navigator.userAgent).browser] = true;
+}
+
+/**
+ * Small JavaScript module for the documentation.
+ */
+var Documentation = {
+
+ init : function() {
+ this.fixFirefoxAnchorBug();
+ this.highlightSearchWords();
+ this.initIndexTable();
+
+ },
+
+ /**
+ * i18n support
+ */
+ TRANSLATIONS : {},
+ PLURAL_EXPR : function(n) { return n == 1 ? 0 : 1; },
+ LOCALE : 'unknown',
+
+ // gettext and ngettext don't access this so that the functions
+ // can safely bound to a different name (_ = Documentation.gettext)
+ gettext : function(string) {
+ var translated = Documentation.TRANSLATIONS[string];
+ if (typeof translated == 'undefined')
+ return string;
+ return (typeof translated == 'string') ? translated : translated[0];
+ },
+
+ ngettext : function(singular, plural, n) {
+ var translated = Documentation.TRANSLATIONS[singular];
+ if (typeof translated == 'undefined')
+ return (n == 1) ? singular : plural;
+ return translated[Documentation.PLURALEXPR(n)];
+ },
+
+ addTranslations : function(catalog) {
+ for (var key in catalog.messages)
+ this.TRANSLATIONS[key] = catalog.messages[key];
+ this.PLURAL_EXPR = new Function('n', 'return +(' + catalog.plural_expr + ')');
+ this.LOCALE = catalog.locale;
+ },
+
+ /**
+ * add context elements like header anchor links
+ */
+ addContextElements : function() {
+ $('div[id] > :header:first').each(function() {
+ $('<a class="headerlink">\u00B6</a>').
+ attr('href', '#' + this.id).
+ attr('title', _('Permalink to this headline')).
+ appendTo(this);
+ });
+ $('dt[id]').each(function() {
+ $('<a class="headerlink">\u00B6</a>').
+ attr('href', '#' + this.id).
+ attr('title', _('Permalink to this definition')).
+ appendTo(this);
+ });
+ },
+
+ /**
+ * workaround a firefox stupidity
+ * see: https://bugzilla.mozilla.org/show_bug.cgi?id=645075
+ */
+ fixFirefoxAnchorBug : function() {
+ if (document.location.hash)
+ window.setTimeout(function() {
+ document.location.href += '';
+ }, 10);
+ },
+
+ /**
+ * highlight the search words provided in the url in the text
+ */
+ highlightSearchWords : function() {
+ var params = $.getQueryParameters();
+ var terms = (params.highlight) ? params.highlight[0].split(/\s+/) : [];
+ if (terms.length) {
+ var body = $('div.body');
+ if (!body.length) {
+ body = $('body');
+ }
+ window.setTimeout(function() {
+ $.each(terms, function() {
+ body.highlightText(this.toLowerCase(), 'highlighted');
+ });
+ }, 10);
+ $('<p class="highlight-link"><a href="javascript:Documentation.' +
+ 'hideSearchWords()">' + _('Hide Search Matches') + '</a></p>')
+ .appendTo($('#searchbox'));
+ }
+ },
+
+ /**
+ * init the domain index toggle buttons
+ */
+ initIndexTable : function() {
+ var togglers = $('img.toggler').click(function() {
+ var src = $(this).attr('src');
+ var idnum = $(this).attr('id').substr(7);
+ $('tr.cg-' + idnum).toggle();
+ if (src.substr(-9) == 'minus.png')
+ $(this).attr('src', src.substr(0, src.length-9) + 'plus.png');
+ else
+ $(this).attr('src', src.substr(0, src.length-8) + 'minus.png');
+ }).css('display', '');
+ if (DOCUMENTATION_OPTIONS.COLLAPSE_INDEX) {
+ togglers.click();
+ }
+ },
+
+ /**
+ * helper function to hide the search marks again
+ */
+ hideSearchWords : function() {
+ $('#searchbox .highlight-link').fadeOut(300);
+ $('span.highlighted').removeClass('highlighted');
+ },
+
+ /**
+ * make the url absolute
+ */
+ makeURL : function(relativeURL) {
+ return DOCUMENTATION_OPTIONS.URL_ROOT + '/' + relativeURL;
+ },
+
+ /**
+ * get the current relative url
+ */
+ getCurrentURL : function() {
+ var path = document.location.pathname;
+ var parts = path.split(/\//);
+ $.each(DOCUMENTATION_OPTIONS.URL_ROOT.split(/\//), function() {
+ if (this == '..')
+ parts.pop();
+ });
+ var url = parts.join('/');
+ return path.substring(url.lastIndexOf('/') + 1, path.length - 1);
+ },
+
+ initOnKeyListeners: function() {
+ $(document).keyup(function(event) {
+ var activeElementType = document.activeElement.tagName;
+ // don't navigate when in search box or textarea
+ if (activeElementType !== 'TEXTAREA' && activeElementType !== 'INPUT' && activeElementType !== 'SELECT') {
+ switch (event.keyCode) {
+ case 37: // left
+ var prevHref = $('link[rel="prev"]').prop('href');
+ if (prevHref) {
+ window.location.href = prevHref;
+ return false;
+ }
+ case 39: // right
+ var nextHref = $('link[rel="next"]').prop('href');
+ if (nextHref) {
+ window.location.href = nextHref;
+ return false;
+ }
+ }
+ }
+ });
+ }
+};
+
+// quick alias for translations
+_ = Documentation.gettext;
+
+$(document).ready(function() {
+ Documentation.init();
+});
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/down-pressed.png
----------------------------------------------------------------------
diff --git a/_static/down-pressed.png b/_static/down-pressed.png
new file mode 100644
index 0000000..7c30d00
Binary files /dev/null and b/_static/down-pressed.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/down.png
----------------------------------------------------------------------
diff --git a/_static/down.png b/_static/down.png
new file mode 100644
index 0000000..f48098a
Binary files /dev/null and b/_static/down.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/file.png
----------------------------------------------------------------------
diff --git a/_static/file.png b/_static/file.png
new file mode 100644
index 0000000..254c60b
Binary files /dev/null and b/_static/file.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/fonts/Inconsolata-Bold.ttf
----------------------------------------------------------------------
diff --git a/_static/fonts/Inconsolata-Bold.ttf b/_static/fonts/Inconsolata-Bold.ttf
new file mode 100644
index 0000000..58c9fef
Binary files /dev/null and b/_static/fonts/Inconsolata-Bold.ttf differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/fonts/Inconsolata-Regular.ttf
----------------------------------------------------------------------
diff --git a/_static/fonts/Inconsolata-Regular.ttf b/_static/fonts/Inconsolata-Regular.ttf
new file mode 100644
index 0000000..a87ffba
Binary files /dev/null and b/_static/fonts/Inconsolata-Regular.ttf differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/fonts/Lato-Bold.ttf
----------------------------------------------------------------------
diff --git a/_static/fonts/Lato-Bold.ttf b/_static/fonts/Lato-Bold.ttf
new file mode 100644
index 0000000..7434369
Binary files /dev/null and b/_static/fonts/Lato-Bold.ttf differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/fonts/Lato-Regular.ttf
----------------------------------------------------------------------
diff --git a/_static/fonts/Lato-Regular.ttf b/_static/fonts/Lato-Regular.ttf
new file mode 100644
index 0000000..04ea8ef
Binary files /dev/null and b/_static/fonts/Lato-Regular.ttf differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/fonts/RobotoSlab-Bold.ttf
----------------------------------------------------------------------
diff --git a/_static/fonts/RobotoSlab-Bold.ttf b/_static/fonts/RobotoSlab-Bold.ttf
new file mode 100644
index 0000000..df5d1df
Binary files /dev/null and b/_static/fonts/RobotoSlab-Bold.ttf differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/fonts/RobotoSlab-Regular.ttf
----------------------------------------------------------------------
diff --git a/_static/fonts/RobotoSlab-Regular.ttf b/_static/fonts/RobotoSlab-Regular.ttf
new file mode 100644
index 0000000..eb52a79
Binary files /dev/null and b/_static/fonts/RobotoSlab-Regular.ttf differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/fonts/fontawesome-webfont.eot
----------------------------------------------------------------------
diff --git a/_static/fonts/fontawesome-webfont.eot b/_static/fonts/fontawesome-webfont.eot
new file mode 100644
index 0000000..84677bc
Binary files /dev/null and b/_static/fonts/fontawesome-webfont.eot differ
[27/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/druid_hook.html
----------------------------------------------------------------------
diff --git a/_modules/druid_hook.html b/_modules/druid_hook.html
new file mode 100644
index 0000000..97bb50f
--- /dev/null
+++ b/_modules/druid_hook.html
@@ -0,0 +1,369 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>druid_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>druid_hook</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for druid_hook</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">print_function</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">import</span> <span class="nn">json</span>
+<span class="kn">import</span> <span class="nn">time</span>
+
+<span class="kn">from</span> <span class="nn">pydruid.client</span> <span class="kn">import</span> <span class="n">PyDruid</span>
+<span class="kn">import</span> <span class="nn">requests</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="kn">import</span> <span class="n">BaseHook</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+
+<span class="n">LOAD_CHECK_INTERVAL</span> <span class="o">=</span> <span class="mi">5</span>
+<span class="n">DEFAULT_TARGET_PARTITION_SIZE</span> <span class="o">=</span> <span class="mi">5000000</span>
+
+<span class="k">class</span> <span class="nc">AirflowDruidLoadException</span><span class="p">(</span><span class="n">AirflowException</span><span class="p">):</span>
+ <span class="k">pass</span>
+
+
+<div class="viewcode-block" id="DruidHook"><a class="viewcode-back" href="../code.html#airflow.hooks.DruidHook">[docs]</a><span class="k">class</span> <span class="nc">DruidHook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> Interact with druid.</span>
+<span class="sd"> '''</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">druid_query_conn_id</span><span class="o">=</span><span class="s1">'druid_query_default'</span><span class="p">,</span>
+ <span class="n">druid_ingest_conn_id</span><span class="o">=</span><span class="s1">'druid_ingest_default'</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">druid_query_conn_id</span> <span class="o">=</span> <span class="n">druid_query_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">druid_ingest_conn_id</span> <span class="o">=</span> <span class="n">druid_ingest_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">header</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'content-type'</span><span class="p">:</span> <span class="s1">'application/json'</span><span class="p">}</span>
+
+<div class="viewcode-block" id="DruidHook.get_conn"><a class="viewcode-back" href="../code.html#airflow.hooks.DruidHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a druid connection object for query</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">druid_query_conn_id</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">PyDruid</span><span class="p">(</span>
+ <span class="s2">"http://{conn.host}:{conn.port}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()),</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'endpoint'</span><span class="p">,</span> <span class="s1">''</span><span class="p">))</span></div>
+
+ <span class="nd">@property</span>
+ <span class="k">def</span> <span class="nf">ingest_post_url</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">druid_ingest_conn_id</span><span class="p">)</span>
+ <span class="n">host</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">host</span>
+ <span class="n">port</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">port</span>
+ <span class="n">endpoint</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'endpoint'</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
+ <span class="k">return</span> <span class="s2">"http://{host}:{port}/{endpoint}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
+
+ <span class="k">def</span> <span class="nf">get_ingest_status_url</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">task_id</span><span class="p">):</span>
+ <span class="n">post_url</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">ingest_post_url</span>
+ <span class="k">return</span> <span class="s2">"{post_url}/{task_id}/status"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
+
+<div class="viewcode-block" id="DruidHook.construct_ingest_query"><a class="viewcode-back" href="../code.html#airflow.hooks.DruidHook.construct_ingest_query">[docs]</a> <span class="k">def</span> <span class="nf">construct_ingest_query</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">datasource</span><span class="p">,</span> <span class="n">static_path</span><span class="p">,</span> <span class="n">ts_dim</span><span class="p">,</span> <span class="n">columns</span><span class="p">,</span> <span class="n">metric_spec</span><span class="p">,</span>
+ <span class="n">intervals</span><span class="p">,</span> <span class="n">num_shards</span><span class="p">,</span> <span class="n">target_partition_size</span><span class="p">,</span> <span class="n">hadoop_dependency_coordinates</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Builds an ingest query for an HDFS TSV load.</span>
+
+<span class="sd"> :param datasource: target datasource in druid</span>
+<span class="sd"> :param columns: list of all columns in the TSV, in the right order</span>
+<span class="sd"> """</span>
+
+ <span class="c1"># backward compatibilty for num_shards, but target_partition_size is the default setting</span>
+ <span class="c1"># and overwrites the num_shards</span>
+ <span class="k">if</span> <span class="n">target_partition_size</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">num_shards</span> <span class="o">==</span> <span class="o">-</span><span class="mi">1</span><span class="p">:</span>
+ <span class="n">target_partition_size</span> <span class="o">=</span> <span class="n">DEFAULT_TARGET_PARTITION_SIZE</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">num_shards</span> <span class="o">=</span> <span class="o">-</span><span class="mi">1</span>
+
+ <span class="n">metric_names</span> <span class="o">=</span> <span class="p">[</span>
+ <span class="n">m</span><span class="p">[</span><span class="s1">'fieldName'</span><span class="p">]</span> <span class="k">for</span> <span class="n">m</span> <span class="ow">in</span> <span class="n">metric_spec</span> <span class="k">if</span> <span class="n">m</span><span class="p">[</span><span class="s1">'type'</span><span class="p">]</span> <span class="o">!=</span> <span class="s1">'count'</span><span class="p">]</span>
+ <span class="n">dimensions</span> <span class="o">=</span> <span class="p">[</span><span class="n">c</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">columns</span> <span class="k">if</span> <span class="n">c</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">metric_names</span> <span class="ow">and</span> <span class="n">c</span> <span class="o">!=</span> <span class="n">ts_dim</span><span class="p">]</span>
+ <span class="n">ingest_query_dict</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"index_hadoop"</span><span class="p">,</span>
+ <span class="s2">"spec"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"dataSchema"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"metricsSpec"</span><span class="p">:</span> <span class="n">metric_spec</span><span class="p">,</span>
+ <span class="s2">"granularitySpec"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"queryGranularity"</span><span class="p">:</span> <span class="s2">"NONE"</span><span class="p">,</span>
+ <span class="s2">"intervals"</span><span class="p">:</span> <span class="n">intervals</span><span class="p">,</span>
+ <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"uniform"</span><span class="p">,</span>
+ <span class="s2">"segmentGranularity"</span><span class="p">:</span> <span class="s2">"DAY"</span><span class="p">,</span>
+ <span class="p">},</span>
+ <span class="s2">"parser"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"string"</span><span class="p">,</span>
+ <span class="s2">"parseSpec"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"columns"</span><span class="p">:</span> <span class="n">columns</span><span class="p">,</span>
+ <span class="s2">"dimensionsSpec"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"dimensionExclusions"</span><span class="p">:</span> <span class="p">[],</span>
+ <span class="s2">"dimensions"</span><span class="p">:</span> <span class="n">dimensions</span><span class="p">,</span> <span class="c1"># list of names</span>
+ <span class="s2">"spatialDimensions"</span><span class="p">:</span> <span class="p">[]</span>
+ <span class="p">},</span>
+ <span class="s2">"timestampSpec"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"column"</span><span class="p">:</span> <span class="n">ts_dim</span><span class="p">,</span>
+ <span class="s2">"format"</span><span class="p">:</span> <span class="s2">"auto"</span>
+ <span class="p">},</span>
+ <span class="s2">"format"</span><span class="p">:</span> <span class="s2">"tsv"</span>
+ <span class="p">}</span>
+ <span class="p">},</span>
+ <span class="s2">"dataSource"</span><span class="p">:</span> <span class="n">datasource</span>
+ <span class="p">},</span>
+ <span class="s2">"tuningConfig"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"hadoop"</span><span class="p">,</span>
+ <span class="s2">"jobProperties"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"mapreduce.job.user.classpath.first"</span><span class="p">:</span> <span class="s2">"false"</span><span class="p">,</span>
+ <span class="s2">"mapreduce.map.output.compress"</span><span class="p">:</span> <span class="s2">"false"</span><span class="p">,</span>
+ <span class="s2">"mapreduce.output.fileoutputformat.compress"</span><span class="p">:</span> <span class="s2">"false"</span><span class="p">,</span>
+ <span class="p">},</span>
+ <span class="s2">"partitionsSpec"</span> <span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"type"</span> <span class="p">:</span> <span class="s2">"hashed"</span><span class="p">,</span>
+ <span class="s2">"targetPartitionSize"</span> <span class="p">:</span> <span class="n">target_partition_size</span><span class="p">,</span>
+ <span class="s2">"numShards"</span> <span class="p">:</span> <span class="n">num_shards</span><span class="p">,</span>
+ <span class="p">},</span>
+ <span class="p">},</span>
+ <span class="s2">"ioConfig"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"inputSpec"</span><span class="p">:</span> <span class="p">{</span>
+ <span class="s2">"paths"</span><span class="p">:</span> <span class="n">static_path</span><span class="p">,</span>
+ <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"static"</span>
+ <span class="p">},</span>
+ <span class="s2">"type"</span><span class="p">:</span> <span class="s2">"hadoop"</span>
+ <span class="p">}</span>
+ <span class="p">}</span>
+ <span class="p">}</span>
+ <span class="k">if</span> <span class="n">hadoop_dependency_coordinates</span><span class="p">:</span>
+ <span class="n">ingest_query_dict</span><span class="p">[</span>
+ <span class="s1">'hadoopDependencyCoordinates'</span><span class="p">]</span> <span class="o">=</span> <span class="n">hadoop_dependency_coordinates</span>
+
+ <span class="k">return</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">ingest_query_dict</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">4</span><span class="p">)</span></div>
+
+ <span class="k">def</span> <span class="nf">send_ingest_query</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">datasource</span><span class="p">,</span> <span class="n">static_path</span><span class="p">,</span> <span class="n">ts_dim</span><span class="p">,</span> <span class="n">columns</span><span class="p">,</span> <span class="n">metric_spec</span><span class="p">,</span>
+ <span class="n">intervals</span><span class="p">,</span> <span class="n">num_shards</span><span class="p">,</span> <span class="n">target_partition_size</span><span class="p">,</span> <span class="n">hadoop_dependency_coordinates</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="n">query</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">construct_ingest_query</span><span class="p">(</span>
+ <span class="n">datasource</span><span class="p">,</span> <span class="n">static_path</span><span class="p">,</span> <span class="n">ts_dim</span><span class="p">,</span> <span class="n">columns</span><span class="p">,</span>
+ <span class="n">metric_spec</span><span class="p">,</span> <span class="n">intervals</span><span class="p">,</span> <span class="n">num_shards</span><span class="p">,</span> <span class="n">target_partition_size</span><span class="p">,</span> <span class="n">hadoop_dependency_coordinates</span><span class="p">)</span>
+ <span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">post</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">ingest_post_url</span><span class="p">,</span> <span class="n">headers</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">header</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">query</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">ingest_post_url</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
+ <span class="n">d</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
+ <span class="k">if</span> <span class="s2">"task"</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">d</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowDruidLoadException</span><span class="p">(</span>
+ <span class="s2">"[Error]: Ingesting data to druid failed."</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">d</span><span class="p">[</span><span class="s2">"task"</span><span class="p">]</span>
+
+<div class="viewcode-block" id="DruidHook.load_from_hdfs"><a class="viewcode-back" href="../code.html#airflow.hooks.DruidHook.load_from_hdfs">[docs]</a> <span class="k">def</span> <span class="nf">load_from_hdfs</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">datasource</span><span class="p">,</span> <span class="n">static_path</span><span class="p">,</span> <span class="n">ts_dim</span><span class="p">,</span> <span class="n">columns</span><span class="p">,</span>
+ <span class="n">intervals</span><span class="p">,</span> <span class="n">num_shards</span><span class="p">,</span> <span class="n">target_partition_size</span><span class="p">,</span> <span class="n">metric_spec</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">hadoop_dependency_coordinates</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> load data to druid from hdfs</span>
+<span class="sd"> :params ts_dim: The column name to use as a timestamp</span>
+<span class="sd"> :params metric_spec: A list of dictionaries</span>
+<span class="sd"> """</span>
+ <span class="n">task_id</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">send_ingest_query</span><span class="p">(</span>
+ <span class="n">datasource</span><span class="p">,</span> <span class="n">static_path</span><span class="p">,</span> <span class="n">ts_dim</span><span class="p">,</span> <span class="n">columns</span><span class="p">,</span> <span class="n">metric_spec</span><span class="p">,</span>
+ <span class="n">intervals</span><span class="p">,</span> <span class="n">num_shards</span><span class="p">,</span> <span class="n">target_partition_size</span><span class="p">,</span> <span class="n">hadoop_dependency_coordinates</span><span class="p">)</span>
+ <span class="n">status_url</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_ingest_status_url</span><span class="p">(</span><span class="n">task_id</span><span class="p">)</span>
+ <span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
+ <span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">status_url</span><span class="p">)</span>
+ <span class="n">d</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">text</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">d</span><span class="p">[</span><span class="s1">'status'</span><span class="p">][</span><span class="s1">'status'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'FAILED'</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="n">d</span><span class="p">)</span>
+ <span class="k">raise</span> <span class="n">AirflowDruidLoadException</span><span class="p">(</span>
+ <span class="s2">"[Error]: Ingesting data to druid failed."</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="n">d</span><span class="p">[</span><span class="s1">'status'</span><span class="p">][</span><span class="s1">'status'</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'SUCCESS'</span><span class="p">:</span>
+ <span class="k">break</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">LOAD_CHECK_INTERVAL</span><span class="p">)</span></div></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/dummy_operator.html
----------------------------------------------------------------------
diff --git a/_modules/dummy_operator.html b/_modules/dummy_operator.html
new file mode 100644
index 0000000..25f7c29
--- /dev/null
+++ b/_modules/dummy_operator.html
@@ -0,0 +1,219 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>dummy_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>dummy_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for dummy_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="DummyOperator"><a class="viewcode-back" href="../code.html#airflow.operators.DummyOperator">[docs]</a><span class="k">class</span> <span class="nc">DummyOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Operator that does literally nothing. It can be used to group tasks in a</span>
+<span class="sd"> DAG.</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">()</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#e8f7e4'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">DummyOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="k">pass</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/email_operator.html
----------------------------------------------------------------------
diff --git a/_modules/email_operator.html b/_modules/email_operator.html
new file mode 100644
index 0000000..7c9eeac
--- /dev/null
+++ b/_modules/email_operator.html
@@ -0,0 +1,240 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>email_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>email_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for email_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.email</span> <span class="kn">import</span> <span class="n">send_email</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="EmailOperator"><a class="viewcode-back" href="../code.html#airflow.operators.EmailOperator">[docs]</a><span class="k">class</span> <span class="nc">EmailOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Sends an email.</span>
+
+<span class="sd"> :param to: list of emails to send the email to</span>
+<span class="sd"> :type to: list or string (comma or semicolon delimited)</span>
+<span class="sd"> :param subject: subject line for the email (templated)</span>
+<span class="sd"> :type subject: string</span>
+<span class="sd"> :param html_content: content of the email (templated), html markup</span>
+<span class="sd"> is allowed</span>
+<span class="sd"> :type html_content: string</span>
+<span class="sd"> :param files: file names to attach in email</span>
+<span class="sd"> :type files: list</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'subject'</span><span class="p">,</span> <span class="s1">'html_content'</span><span class="p">)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.html'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#e6faf9'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">to</span><span class="p">,</span>
+ <span class="n">subject</span><span class="p">,</span>
+ <span class="n">html_content</span><span class="p">,</span>
+ <span class="n">files</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">EmailOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">to</span> <span class="o">=</span> <span class="n">to</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">subject</span> <span class="o">=</span> <span class="n">subject</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">html_content</span> <span class="o">=</span> <span class="n">html_content</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">files</span> <span class="o">=</span> <span class="n">files</span> <span class="ow">or</span> <span class="p">[]</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">send_email</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">to</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">subject</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">html_content</span><span class="p">,</span> <span class="n">files</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">files</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/ftp_hook.html
----------------------------------------------------------------------
diff --git a/_modules/ftp_hook.html b/_modules/ftp_hook.html
new file mode 100644
index 0000000..f433866
--- /dev/null
+++ b/_modules/ftp_hook.html
@@ -0,0 +1,427 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>ftp_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>ftp_hook</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for ftp_hook</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="c1">#</span>
+
+<span class="kn">import</span> <span class="nn">datetime</span>
+<span class="kn">import</span> <span class="nn">ftplib</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">import</span> <span class="nn">os.path</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="kn">import</span> <span class="n">BaseHook</span>
+<span class="kn">from</span> <span class="nn">past.builtins</span> <span class="kn">import</span> <span class="nb">basestring</span>
+
+
+<span class="k">def</span> <span class="nf">mlsd</span><span class="p">(</span><span class="n">conn</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="s2">""</span><span class="p">,</span> <span class="n">facts</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> BACKPORT FROM PYTHON3 FTPLIB</span>
+
+<span class="sd"> List a directory in a standardized format by using MLSD</span>
+<span class="sd"> command (RFC-3659). If path is omitted the current directory</span>
+<span class="sd"> is assumed. "facts" is a list of strings representing the type</span>
+<span class="sd"> of information desired (e.g. ["type", "size", "perm"]).</span>
+
+<span class="sd"> Return a generator object yielding a tuple of two elements</span>
+<span class="sd"> for every file found in path.</span>
+<span class="sd"> First element is the file name, the second one is a dictionary</span>
+<span class="sd"> including a variable number of "facts" depending on the server</span>
+<span class="sd"> and whether "facts" argument has been provided.</span>
+<span class="sd"> '''</span>
+ <span class="n">facts</span> <span class="o">=</span> <span class="n">facts</span> <span class="ow">or</span> <span class="p">[]</span>
+ <span class="k">if</span> <span class="n">facts</span><span class="p">:</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">sendcmd</span><span class="p">(</span><span class="s2">"OPTS MLST "</span> <span class="o">+</span> <span class="s2">";"</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">facts</span><span class="p">)</span> <span class="o">+</span> <span class="s2">";"</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">path</span><span class="p">:</span>
+ <span class="n">cmd</span> <span class="o">=</span> <span class="s2">"MLSD </span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span> <span class="n">path</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">cmd</span> <span class="o">=</span> <span class="s2">"MLSD"</span>
+ <span class="n">lines</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">retrlines</span><span class="p">(</span><span class="n">cmd</span><span class="p">,</span> <span class="n">lines</span><span class="o">.</span><span class="n">append</span><span class="p">)</span>
+ <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">lines</span><span class="p">:</span>
+ <span class="n">facts_found</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">name</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">rstrip</span><span class="p">(</span><span class="n">ftplib</span><span class="o">.</span><span class="n">CRLF</span><span class="p">)</span><span class="o">.</span><span class="n">partition</span><span class="p">(</span><span class="s1">' '</span><span class="p">)</span>
+ <span class="n">entry</span> <span class="o">=</span> <span class="p">{}</span>
+ <span class="k">for</span> <span class="n">fact</span> <span class="ow">in</span> <span class="n">facts_found</span><span class="p">[:</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">";"</span><span class="p">):</span>
+ <span class="n">key</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">value</span> <span class="o">=</span> <span class="n">fact</span><span class="o">.</span><span class="n">partition</span><span class="p">(</span><span class="s2">"="</span><span class="p">)</span>
+ <span class="n">entry</span><span class="p">[</span><span class="n">key</span><span class="o">.</span><span class="n">lower</span><span class="p">()]</span> <span class="o">=</span> <span class="n">value</span>
+ <span class="k">yield</span> <span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">entry</span><span class="p">)</span>
+
+
+<div class="viewcode-block" id="FTPHook"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.FTPHook">[docs]</a><span class="k">class</span> <span class="nc">FTPHook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
+
+ <span class="sd">"""</span>
+<span class="sd"> Interact with FTP.</span>
+
+<span class="sd"> Errors that may occur throughout but should be handled</span>
+<span class="sd"> downstream.</span>
+<span class="sd"> """</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">ftp_conn_id</span><span class="o">=</span><span class="s1">'ftp_default'</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">ftp_conn_id</span> <span class="o">=</span> <span class="n">ftp_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="o">=</span> <span class="bp">None</span>
+
+<div class="viewcode-block" id="FTPHook.get_conn"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.FTPHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a FTP connection object</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
+ <span class="n">params</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">ftp_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="o">=</span> <span class="n">ftplib</span><span class="o">.</span><span class="n">FTP</span><span class="p">(</span><span class="n">params</span><span class="o">.</span><span class="n">host</span><span class="p">,</span> <span class="n">params</span><span class="o">.</span><span class="n">login</span><span class="p">,</span> <span class="n">params</span><span class="o">.</span><span class="n">password</span><span class="p">)</span>
+
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span></div>
+
+<div class="viewcode-block" id="FTPHook.close_conn"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.FTPHook.close_conn">[docs]</a> <span class="k">def</span> <span class="nf">close_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Closes the connection. An error will occur if the</span>
+<span class="sd"> connection wasnt ever opened.</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">quit</span><span class="p">()</span></div>
+
+<div class="viewcode-block" id="FTPHook.describe_directory"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.FTPHook.describe_directory">[docs]</a> <span class="k">def</span> <span class="nf">describe_directory</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a dictionary of {filename: {attributes}} for all files</span>
+<span class="sd"> on the remote system (where the MLSD command is supported).</span>
+
+<span class="sd"> :param path: full path to the remote directory</span>
+<span class="sd"> :type path: str</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">cwd</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="c1"># only works in Python 3</span>
+ <span class="n">files</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">mlsd</span><span class="p">())</span>
+ <span class="k">except</span> <span class="ne">AttributeError</span><span class="p">:</span>
+ <span class="n">files</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">mlsd</span><span class="p">(</span><span class="n">conn</span><span class="p">))</span>
+ <span class="k">return</span> <span class="n">files</span></div>
+
+<div class="viewcode-block" id="FTPHook.list_directory"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.FTPHook.list_directory">[docs]</a> <span class="k">def</span> <span class="nf">list_directory</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">,</span> <span class="n">nlst</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a list of files on the remote system.</span>
+
+<span class="sd"> :param path: full path to the remote directory to list</span>
+<span class="sd"> :type path: str</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">cwd</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
+
+ <span class="n">files</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">nlst</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">files</span></div>
+
+<div class="viewcode-block" id="FTPHook.create_directory"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.FTPHook.create_directory">[docs]</a> <span class="k">def</span> <span class="nf">create_directory</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Creates a directory on the remote system.</span>
+
+<span class="sd"> :param path: full path to the remote directory to create</span>
+<span class="sd"> :type path: str</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">mkd</span><span class="p">(</span><span class="n">path</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="FTPHook.delete_directory"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.FTPHook.delete_directory">[docs]</a> <span class="k">def</span> <span class="nf">delete_directory</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Deletes a directory on the remote system.</span>
+
+<span class="sd"> :param path: full path to the remote directory to delete</span>
+<span class="sd"> :type path: str</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">rmd</span><span class="p">(</span><span class="n">path</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="FTPHook.retrieve_file"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.FTPHook.retrieve_file">[docs]</a> <span class="k">def</span> <span class="nf">retrieve_file</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">remote_full_path</span><span class="p">,</span> <span class="n">local_full_path_or_buffer</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Transfers the remote file to a local location.</span>
+
+<span class="sd"> If local_full_path_or_buffer is a string path, the file will be put</span>
+<span class="sd"> at that location; if it is a file-like buffer, the file will</span>
+<span class="sd"> be written to the buffer but not closed.</span>
+
+<span class="sd"> :param remote_full_path: full path to the remote file</span>
+<span class="sd"> :type remote_full_path: str</span>
+<span class="sd"> :param local_full_path_or_buffer: full path to the local file or a</span>
+<span class="sd"> file-like buffer</span>
+<span class="sd"> :type local_full_path: str or file-like buffer</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+
+ <span class="n">is_path</span> <span class="o">=</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">local_full_path_or_buffer</span><span class="p">,</span> <span class="nb">basestring</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="n">is_path</span><span class="p">:</span>
+ <span class="n">output_handle</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">local_full_path_or_buffer</span><span class="p">,</span> <span class="s1">'wb'</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">output_handle</span> <span class="o">=</span> <span class="n">local_full_path_or_buffer</span>
+
+ <span class="n">remote_path</span><span class="p">,</span> <span class="n">remote_file_name</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">remote_full_path</span><span class="p">)</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">cwd</span><span class="p">(</span><span class="n">remote_path</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Retrieving file from FTP: {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">remote_full_path</span><span class="p">))</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">retrbinary</span><span class="p">(</span><span class="s1">'RETR </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">remote_file_name</span><span class="p">,</span> <span class="n">output_handle</span><span class="o">.</span><span class="n">write</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Finished etrieving file from FTP: {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">remote_full_path</span><span class="p">))</span>
+
+ <span class="k">if</span> <span class="n">is_path</span><span class="p">:</span>
+ <span class="n">output_handle</span><span class="o">.</span><span class="n">close</span><span class="p">()</span></div>
+
+<div class="viewcode-block" id="FTPHook.store_file"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.FTPHook.store_file">[docs]</a> <span class="k">def</span> <span class="nf">store_file</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">remote_full_path</span><span class="p">,</span> <span class="n">local_full_path_or_buffer</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Transfers a local file to the remote location.</span>
+
+<span class="sd"> If local_full_path_or_buffer is a string path, the file will be read</span>
+<span class="sd"> from that location; if it is a file-like buffer, the file will</span>
+<span class="sd"> be read from the buffer but not closed.</span>
+
+<span class="sd"> :param remote_full_path: full path to the remote file</span>
+<span class="sd"> :type remote_full_path: str</span>
+<span class="sd"> :param local_full_path_or_buffer: full path to the local file or a</span>
+<span class="sd"> file-like buffer</span>
+<span class="sd"> :type local_full_path_or_buffer: str or file-like buffer</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+
+ <span class="n">is_path</span> <span class="o">=</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">local_full_path_or_buffer</span><span class="p">,</span> <span class="nb">basestring</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="n">is_path</span><span class="p">:</span>
+ <span class="n">input_handle</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">local_full_path_or_buffer</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">input_handle</span> <span class="o">=</span> <span class="n">local_full_path_or_buffer</span>
+ <span class="n">remote_path</span><span class="p">,</span> <span class="n">remote_file_name</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">remote_full_path</span><span class="p">)</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">cwd</span><span class="p">(</span><span class="n">remote_path</span><span class="p">)</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">storbinary</span><span class="p">(</span><span class="s1">'STOR </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="n">remote_file_name</span><span class="p">,</span> <span class="n">input_handle</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="n">is_path</span><span class="p">:</span>
+ <span class="n">input_handle</span><span class="o">.</span><span class="n">close</span><span class="p">()</span></div>
+
+<div class="viewcode-block" id="FTPHook.delete_file"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.FTPHook.delete_file">[docs]</a> <span class="k">def</span> <span class="nf">delete_file</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Removes a file on the FTP Server</span>
+
+<span class="sd"> :param path: full path to the remote file</span>
+<span class="sd"> :type path: str</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="n">path</span><span class="p">)</span></div>
+
+ <span class="k">def</span> <span class="nf">get_mod_time</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">path</span><span class="p">):</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">ftp_mdtm</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">sendcmd</span><span class="p">(</span><span class="s1">'MDTM '</span> <span class="o">+</span> <span class="n">path</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">ftp_mdtm</span><span class="p">[</span><span class="mi">4</span><span class="p">:],</span> <span class="s1">'%Y%m</span><span class="si">%d</span><span class="s1">%H%M%S'</span><span class="p">)</span></div>
+
+
+<span class="k">class</span> <span class="nc">FTPSHook</span><span class="p">(</span><span class="n">FTPHook</span><span class="p">):</span>
+
+ <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a FTPS connection object</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
+ <span class="n">params</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">ftp_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="o">=</span> <span class="n">ftplib</span><span class="o">.</span><span class="n">FTP_TLS</span><span class="p">(</span>
+ <span class="n">params</span><span class="o">.</span><span class="n">host</span><span class="p">,</span> <span class="n">params</span><span class="o">.</span><span class="n">login</span><span class="p">,</span> <span class="n">params</span><span class="o">.</span><span class="n">password</span>
+ <span class="p">)</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[05/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/concepts.html
----------------------------------------------------------------------
diff --git a/concepts.html b/concepts.html
new file mode 100644
index 0000000..871c608
--- /dev/null
+++ b/concepts.html
@@ -0,0 +1,897 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Concepts — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="Data Profiling" href="profiling.html"/>
+ <link rel="prev" title="UI / Screenshots" href="ui.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">Concepts</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#core-ideas">Core Ideas</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#dags">DAGs</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#scope">Scope</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#default-arguments">Default Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#context-manager">Context Manager</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#operators">Operators</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="#dag-assignment">DAG Assignment</a></li>
+<li class="toctree-l4"><a class="reference internal" href="#bitshift-composition">Bitshift Composition</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="#tasks">Tasks</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#task-instances">Task Instances</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#workflows">Workflows</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="#additional-functionality">Additional Functionality</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#hooks">Hooks</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#pools">Pools</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#connections">Connections</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#queues">Queues</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#xcoms">XComs</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#variables">Variables</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#branching">Branching</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#subdags">SubDAGs</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#slas">SLAs</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#trigger-rules">Trigger Rules</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#zombies-undeads">Zombies & Undeads</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#cluster-policy">Cluster Policy</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#documentation-notes">Documentation & Notes</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#jinja-templating">Jinja Templating</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="#packaged-dags">Packaged dags</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>Concepts</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/concepts.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="concepts">
+<h1>Concepts<a class="headerlink" href="#concepts" title="Permalink to this headline">�</a></h1>
+<p>The Airflow Platform is a tool for describing, executing, and monitoring
+workflows.</p>
+<div class="section" id="core-ideas">
+<h2>Core Ideas<a class="headerlink" href="#core-ideas" title="Permalink to this headline">�</a></h2>
+<div class="section" id="dags">
+<h3>DAGs<a class="headerlink" href="#dags" title="Permalink to this headline">�</a></h3>
+<p>In Airflow, a <code class="docutils literal"><span class="pre">DAG</span></code> – or a Directed Acyclic Graph – is a collection of all
+the tasks you want to run, organized in a way that reflects their relationships
+and dependencies.</p>
+<p>For example, a simple DAG could consist of three tasks: A, B, and C. It could
+say that A has to run successfully before B can run, but C can run anytime. It
+could say that task A times out after 5 minutes, and B can be restarted up to 5
+times in case it fails. It might also say that the workflow will run every night
+at 10pm, but shouldn’t start until a certain date.</p>
+<p>In this way, a DAG describes <em>how</em> you want to carry out your workflow; but
+notice that we haven’t said anything about <em>what</em> we actually want to do! A, B,
+and C could be anything. Maybe A prepares data for B to analyze while C sends an
+email. Or perhaps A monitors your location so B can open your garage door while
+C turns on your house lights. The important thing is that the DAG isn’t
+concerned with what its constituent tasks do; its job is to make sure that
+whatever they do happens at the right time, or in the right order, or with the
+right handling of any unexpected issues.</p>
+<p>DAGs are defined in standard Python files that are placed in Airflow’s
+<code class="docutils literal"><span class="pre">DAG_FOLDER</span></code>. Airflow will execute the code in each file to dynamically build
+the <code class="docutils literal"><span class="pre">DAG</span></code> objects. You can have as many DAGs as you want, each describing an
+arbitrary number of tasks. In general, each one should correspond to a single
+logical workflow.</p>
+<div class="section" id="scope">
+<h4>Scope<a class="headerlink" href="#scope" title="Permalink to this headline">�</a></h4>
+<p>Airflow will load any <code class="docutils literal"><span class="pre">DAG</span></code> object it can import from a DAGfile. Critically,
+that means the DAG must appear in <code class="docutils literal"><span class="pre">globals()</span></code>. Consider the following two
+DAGs. Only <code class="docutils literal"><span class="pre">dag_1</span></code> will be loaded; the other one only appears in a local
+scope.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">dag_1</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'this_dag_will_be_discovered'</span><span class="p">)</span>
+
+<span class="k">def</span> <span class="nf">my_function</span><span class="p">()</span>
+ <span class="n">dag_2</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'but_this_dag_will_not'</span><span class="p">)</span>
+
+<span class="n">my_function</span><span class="p">()</span>
+</pre></div>
+</div>
+<p>Sometimes this can be put to good use. For example, a common pattern with
+<code class="docutils literal"><span class="pre">SubDagOperator</span></code> is to define the subdag inside a function so that Airflow
+doesn’t try to load it as a standalone DAG.</p>
+</div>
+<div class="section" id="default-arguments">
+<h4>Default Arguments<a class="headerlink" href="#default-arguments" title="Permalink to this headline">�</a></h4>
+<p>If a dictionary of <code class="docutils literal"><span class="pre">default_args</span></code> is passed to a DAG, it will apply them to
+any of its operators. This makes it easy to apply a common parameter to many operators without having to type it many times.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">default_args</span><span class="o">=</span><span class="nb">dict</span><span class="p">(</span>
+ <span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2016</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">),</span>
+ <span class="n">owner</span><span class="o">=</span><span class="s1">'Airflow'</span><span class="p">)</span>
+
+<span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'my_dag'</span><span class="p">,</span> <span class="n">default_args</span><span class="o">=</span><span class="n">default_args</span><span class="p">)</span>
+<span class="n">op</span> <span class="o">=</span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'dummy'</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+<span class="nb">print</span><span class="p">(</span><span class="n">op</span><span class="o">.</span><span class="n">owner</span><span class="p">)</span> <span class="c1"># Airflow</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="context-manager">
+<h4>Context Manager<a class="headerlink" href="#context-manager" title="Permalink to this headline">�</a></h4>
+<p><em>Added in Airflow 1.8</em></p>
+<p>DAGs can be used as context managers to automatically assign new operators to that DAG.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'my_dag'</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2016</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
+ <span class="n">op</span> <span class="o">=</span> <span class="n">DummyOperator</span><span class="p">(</span><span class="s1">'op'</span><span class="p">)</span>
+
+<span class="n">op</span><span class="o">.</span><span class="n">dag</span> <span class="ow">is</span> <span class="n">dag</span> <span class="c1"># True</span>
+</pre></div>
+</div>
+</div>
+</div>
+<div class="section" id="operators">
+<h3>Operators<a class="headerlink" href="#operators" title="Permalink to this headline">�</a></h3>
+<p>While DAGs describe <em>how</em> to run a workflow, <code class="docutils literal"><span class="pre">Operators</span></code> determine what
+actually gets done.</p>
+<p>An operator describes a single task in a workflow. Operators are usually (but
+not always) atomic, meaning they can stand on their own and don’t need to share
+resources with any other operators. The DAG will make sure that operators run in
+the correct certain order; other than those dependencies, operators generally
+run independently. In fact, they may run on two completely different machines.</p>
+<p>This is a subtle but very important point: in general, if two operators need to
+share information, like a filename or small amount of data, you should consider
+combining them into a single operator. If it absolutely can’t be avoided,
+Airflow does have a feature for operator cross-communication called XCom that is
+described elsewhere in this document.</p>
+<p>Airflow provides operators for many common tasks, including:</p>
+<ul class="simple">
+<li><code class="docutils literal"><span class="pre">BashOperator</span></code> - executes a bash command</li>
+<li><code class="docutils literal"><span class="pre">PythonOperator</span></code> - calls an arbitrary Python function</li>
+<li><code class="docutils literal"><span class="pre">EmailOperator</span></code> - sends an email</li>
+<li><code class="docutils literal"><span class="pre">HTTPOperator</span></code> - sends an HTTP request</li>
+<li><code class="docutils literal"><span class="pre">SqlOperator</span></code> - executes a SQL command</li>
+<li><code class="docutils literal"><span class="pre">Sensor</span></code> - waits for a certain time, file, database row, S3 key, etc...</li>
+</ul>
+<p>In addition to these basic building blocks, there are many more specific
+operators: <code class="docutils literal"><span class="pre">DockerOperator</span></code>, <code class="docutils literal"><span class="pre">HiveOperator</span></code>, <code class="docutils literal"><span class="pre">S3FileTransferOperator</span></code>,
+<code class="docutils literal"><span class="pre">PrestoToMysqlOperator</span></code>, <code class="docutils literal"><span class="pre">SlackOperator</span></code>... you get the idea!</p>
+<p>The <code class="docutils literal"><span class="pre">airflow/contrib/</span></code> directory contains yet more operators built by the
+community. These operators aren’t always as complete or well-tested as those in
+the main distribution, but allow users to more easily add new functionality to
+the platform.</p>
+<p>Operators are only loaded by Airflow if they are assigned to a DAG.</p>
+<div class="section" id="dag-assignment">
+<h4>DAG Assignment<a class="headerlink" href="#dag-assignment" title="Permalink to this headline">�</a></h4>
+<p><em>Added in Airflow 1.8</em></p>
+<p>Operators do not have to be assigned to DAGs immediately (previously <code class="docutils literal"><span class="pre">dag</span></code> was
+a required argument). However, once an operator is assigned to a DAG, it can not
+be transferred or unassigned. DAG assignment can be done explicitly when the
+operator is created, through deferred assignment, or even inferred from other
+operators.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'my_dag'</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2016</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span>
+
+<span class="c1"># sets the DAG explicitly</span>
+<span class="n">explicit_op</span> <span class="o">=</span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'op1'</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+
+<span class="c1"># deferred DAG assignment</span>
+<span class="n">deferred_op</span> <span class="o">=</span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'op2'</span><span class="p">)</span>
+<span class="n">deferred_op</span><span class="o">.</span><span class="n">dag</span> <span class="o">=</span> <span class="n">dag</span>
+
+<span class="c1"># inferred DAG assignment (linked operators must be in the same DAG)</span>
+<span class="n">inferred_op</span> <span class="o">=</span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'op3'</span><span class="p">)</span>
+<span class="n">inferred_op</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">deferred_op</span><span class="p">)</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="bitshift-composition">
+<h4>Bitshift Composition<a class="headerlink" href="#bitshift-composition" title="Permalink to this headline">�</a></h4>
+<p><em>Added in Airflow 1.8</em></p>
+<p>Traditionally, operator relationships are set with the <code class="docutils literal"><span class="pre">set_upstream()</span></code> and
+<code class="docutils literal"><span class="pre">set_downstream()</span></code> methods. In Airflow 1.8, this can be done with the Python
+bitshift operators <code class="docutils literal"><span class="pre">>></span></code> and <code class="docutils literal"><span class="pre"><<</span></code>. The following four statements are all
+functionally equivalent:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">op1</span> <span class="o">>></span> <span class="n">op2</span>
+<span class="n">op1</span><span class="o">.</span><span class="n">set_downstream</span><span class="p">(</span><span class="n">op2</span><span class="p">)</span>
+
+<span class="n">op2</span> <span class="o"><<</span> <span class="n">op1</span>
+<span class="n">op2</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">op1</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>When using the bitshift to compose operators, the relationship is set in the
+direction that the bitshift operator points. For example, <code class="docutils literal"><span class="pre">op1</span> <span class="pre">>></span> <span class="pre">op2</span></code> means
+that <code class="docutils literal"><span class="pre">op1</span></code> runs first and <code class="docutils literal"><span class="pre">op2</span></code> runs second. Multiple operators can be
+composed – keep in mind the chain is executed left-to-right and the rightmost
+object is always returned. For example:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">op1</span> <span class="o">>></span> <span class="n">op2</span> <span class="o">>></span> <span class="n">op3</span> <span class="o"><<</span> <span class="n">op4</span>
+</pre></div>
+</div>
+<p>is equivalent to:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">op1</span><span class="o">.</span><span class="n">set_downstream</span><span class="p">(</span><span class="n">op2</span><span class="p">)</span>
+<span class="n">op2</span><span class="o">.</span><span class="n">set_downstream</span><span class="p">(</span><span class="n">op3</span><span class="p">)</span>
+<span class="n">op3</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">op4</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>For convenience, the bitshift operators can also be used with DAGs. For example:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">dag</span> <span class="o">>></span> <span class="n">op1</span> <span class="o">>></span> <span class="n">op2</span>
+</pre></div>
+</div>
+<p>is equivalent to:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">op1</span><span class="o">.</span><span class="n">dag</span> <span class="o">=</span> <span class="n">dag</span>
+<span class="n">op1</span><span class="o">.</span><span class="n">set_downstream</span><span class="p">(</span><span class="n">op2</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>We can put this all together to build a simple pipeline:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'my_dag'</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2016</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span>
+ <span class="p">(</span>
+ <span class="n">dag</span>
+ <span class="o">>></span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'dummy_1'</span><span class="p">)</span>
+ <span class="o">>></span> <span class="n">BashOperator</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="s1">'bash_1'</span><span class="p">,</span>
+ <span class="n">bash_command</span><span class="o">=</span><span class="s1">'echo "HELLO!"'</span><span class="p">)</span>
+ <span class="o">>></span> <span class="n">PythonOperator</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="s1">'python_1'</span><span class="p">,</span>
+ <span class="n">python_callable</span><span class="o">=</span><span class="k">lambda</span><span class="p">:</span> <span class="nb">print</span><span class="p">(</span><span class="s2">"GOODBYE!"</span><span class="p">))</span>
+ <span class="p">)</span>
+</pre></div>
+</div>
+</div>
+</div>
+<div class="section" id="tasks">
+<h3>Tasks<a class="headerlink" href="#tasks" title="Permalink to this headline">�</a></h3>
+<p>Once an operator is instantiated, it is referred to as a “task”. The
+instantiation defines specific values when calling the abstract operator, and
+the parameterized task becomes a node in a DAG.</p>
+</div>
+<div class="section" id="task-instances">
+<h3>Task Instances<a class="headerlink" href="#task-instances" title="Permalink to this headline">�</a></h3>
+<p>A task instance represents a specific run of a task and is characterized as the
+combination of a dag, a task, and a point in time. Task instances also have an
+indicative state, which could be “running”, “success”, “failed”, “skipped”, “up
+for retry”, etc.</p>
+</div>
+<div class="section" id="workflows">
+<h3>Workflows<a class="headerlink" href="#workflows" title="Permalink to this headline">�</a></h3>
+<p>You’re now familiar with the core building blocks of Airflow.
+Some of the concepts may sound very similar, but the vocabulary can
+be conceptualized like this:</p>
+<ul class="simple">
+<li>DAG: a description of the order in which work should take place</li>
+<li>Operator: a class that acts as a template for carrying out some work</li>
+<li>Task: a parameterized instance of an operator</li>
+<li>Task Instance: a task that 1) has been assigned to a DAG and 2) has a
+state associated with a specific run of the DAG</li>
+</ul>
+<p>By combining <code class="docutils literal"><span class="pre">DAGs</span></code> and <code class="docutils literal"><span class="pre">Operators</span></code> to create <code class="docutils literal"><span class="pre">TaskInstances</span></code>, you can
+build complex workflows.</p>
+</div>
+</div>
+<div class="section" id="additional-functionality">
+<h2>Additional Functionality<a class="headerlink" href="#additional-functionality" title="Permalink to this headline">�</a></h2>
+<p>In addition to the core Airflow objects, there are a number of more complex
+features that enable behaviors like limiting simultaneous access to resources,
+cross-communication, conditional execution, and more.</p>
+<div class="section" id="hooks">
+<h3>Hooks<a class="headerlink" href="#hooks" title="Permalink to this headline">�</a></h3>
+<p>Hooks are interfaces to external platforms and databases like Hive, S3,
+MySQL, Postgres, HDFS, and Pig. Hooks implement a common interface when
+possible, and act as a building block for operators. They also use
+the <code class="docutils literal"><span class="pre">airflow.models.Connection</span></code> model to retrieve hostnames
+and authentication information. Hooks keep authentication code and
+information out of pipelines, centralized in the metadata database.</p>
+<p>Hooks are also very useful on their own to use in Python scripts,
+Airflow airflow.operators.PythonOperator, and in interactive environments
+like iPython or Jupyter Notebook.</p>
+</div>
+<div class="section" id="pools">
+<h3>Pools<a class="headerlink" href="#pools" title="Permalink to this headline">�</a></h3>
+<p>Some systems can get overwhelmed when too many processes hit them at the same
+time. Airflow pools can be used to <strong>limit the execution parallelism</strong> on
+arbitrary sets of tasks. The list of pools is managed in the UI
+(<code class="docutils literal"><span class="pre">Menu</span> <span class="pre">-></span> <span class="pre">Admin</span> <span class="pre">-></span> <span class="pre">Pools</span></code>) by giving the pools a name and assigning
+it a number of worker slots. Tasks can then be associated with
+one of the existing pools by using the <code class="docutils literal"><span class="pre">pool</span></code> parameter when
+creating tasks (i.e., instantiating operators).</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="n">aggregate_db_message_job</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="s1">'aggregate_db_message_job'</span><span class="p">,</span>
+ <span class="n">execution_timeout</span><span class="o">=</span><span class="n">timedelta</span><span class="p">(</span><span class="n">hours</span><span class="o">=</span><span class="mi">3</span><span class="p">),</span>
+ <span class="n">pool</span><span class="o">=</span><span class="s1">'ep_data_pipeline_db_msg_agg'</span><span class="p">,</span>
+ <span class="n">bash_command</span><span class="o">=</span><span class="n">aggregate_db_message_job_cmd</span><span class="p">,</span>
+ <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+<span class="n">aggregate_db_message_job</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">wait_for_empty_queue</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>The <code class="docutils literal"><span class="pre">pool</span></code> parameter can
+be used in conjunction with <code class="docutils literal"><span class="pre">priority_weight</span></code> to define priorities
+in the queue, and which tasks get executed first as slots open up in the
+pool. The default <code class="docutils literal"><span class="pre">priority_weight</span></code> is <code class="docutils literal"><span class="pre">1</span></code>, and can be bumped to any
+number. When sorting the queue to evaluate which task should be executed
+next, we use the <code class="docutils literal"><span class="pre">priority_weight</span></code>, summed up with all of the
+<code class="docutils literal"><span class="pre">priority_weight</span></code> values from tasks downstream from this task. You can
+use this to bump a specific important task and the whole path to that task
+gets prioritized accordingly.</p>
+<p>Tasks will be scheduled as usual while the slots fill up. Once capacity is
+reached, runnable tasks get queued and their state will show as such in the
+UI. As slots free up, queued tasks start running based on the
+<code class="docutils literal"><span class="pre">priority_weight</span></code> (of the task and its descendants).</p>
+<p>Note that by default tasks aren’t assigned to any pool and their
+execution parallelism is only limited to the executor’s setting.</p>
+</div>
+<div class="section" id="connections">
+<h3>Connections<a class="headerlink" href="#connections" title="Permalink to this headline">�</a></h3>
+<p>The connection information to external systems is stored in the Airflow
+metadata database and managed in the UI (<code class="docutils literal"><span class="pre">Menu</span> <span class="pre">-></span> <span class="pre">Admin</span> <span class="pre">-></span> <span class="pre">Connections</span></code>)
+A <code class="docutils literal"><span class="pre">conn_id</span></code> is defined there and hostname / login / password / schema
+information attached to it. Airflow pipelines can simply refer to the
+centrally managed <code class="docutils literal"><span class="pre">conn_id</span></code> without having to hard code any of this
+information anywhere.</p>
+<p>Many connections with the same <code class="docutils literal"><span class="pre">conn_id</span></code> can be defined and when that
+is the case, and when the <strong>hooks</strong> uses the <code class="docutils literal"><span class="pre">get_connection</span></code> method
+from <code class="docutils literal"><span class="pre">BaseHook</span></code>, Airflow will choose one connection randomly, allowing
+for some basic load balancing and fault tolerance when used in conjunction
+with retries.</p>
+<p>Airflow also has the ability to reference connections via environment
+variables from the operating system. The environment variable needs to be
+prefixed with <code class="docutils literal"><span class="pre">AIRFLOW_CONN_</span></code> to be considered a connection. When
+referencing the connection in the Airflow pipeline, the <code class="docutils literal"><span class="pre">conn_id</span></code> should
+be the name of the variable without the prefix. For example, if the <code class="docutils literal"><span class="pre">conn_id</span></code>
+is named <code class="docutils literal"><span class="pre">POSTGRES_MASTER</span></code> the environment variable should be named
+<code class="docutils literal"><span class="pre">AIRFLOW_CONN_POSTGRES_MASTER</span></code>. Airflow assumes the value returned
+from the environment variable to be in a URI format
+(e.g. <code class="docutils literal"><span class="pre">postgres://user:password@localhost:5432/master</span></code>).</p>
+</div>
+<div class="section" id="queues">
+<h3>Queues<a class="headerlink" href="#queues" title="Permalink to this headline">�</a></h3>
+<p>When using the CeleryExecutor, the celery queues that tasks are sent to
+can be specified. <code class="docutils literal"><span class="pre">queue</span></code> is an attribute of BaseOperator, so any
+task can be assigned to any queue. The default queue for the environment
+is defined in the <code class="docutils literal"><span class="pre">airflow.cfg</span></code>‘s <code class="docutils literal"><span class="pre">celery</span> <span class="pre">-></span> <span class="pre">default_queue</span></code>. This defines
+the queue that tasks get assigned to when not specified, as well as which
+queue Airflow workers listen to when started.</p>
+<p>Workers can listen to one or multiple queues of tasks. When a worker is
+started (using the command <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">worker</span></code>), a set of comma delimited
+queue names can be specified (e.g. <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">worker</span> <span class="pre">-q</span> <span class="pre">spark</span></code>). This worker
+will then only pick up tasks wired to the specified queue(s).</p>
+<p>This can be useful if you need specialized workers, either from a
+resource perspective (for say very lightweight tasks where one worker
+could take thousands of tasks without a problem), or from an environment
+perspective (you want a worker running from within the Spark cluster
+itself because it needs a very specific environment and security rights).</p>
+</div>
+<div class="section" id="xcoms">
+<h3>XComs<a class="headerlink" href="#xcoms" title="Permalink to this headline">�</a></h3>
+<p>XComs let tasks exchange messages, allowing more nuanced forms of control and
+shared state. The name is an abbreviation of “cross-communication”. XComs are
+principally defined by a key, value, and timestamp, but also track attributes
+like the task/DAG that created the XCom and when it should become visible. Any
+object that can be pickled can be used as an XCom value, so users should make
+sure to use objects of appropriate size.</p>
+<p>XComs can be “pushed” (sent) or “pulled” (received). When a task pushes an
+XCom, it makes it generally available to other tasks. Tasks can push XComs at
+any time by calling the <code class="docutils literal"><span class="pre">xcom_push()</span></code> method. In addition, if a task returns
+a value (either from its Operator’s <code class="docutils literal"><span class="pre">execute()</span></code> method, or from a
+PythonOperator’s <code class="docutils literal"><span class="pre">python_callable</span></code> function), then an XCom containing that
+value is automatically pushed.</p>
+<p>Tasks call <code class="docutils literal"><span class="pre">xcom_pull()</span></code> to retrieve XComs, optionally applying filters
+based on criteria like <code class="docutils literal"><span class="pre">key</span></code>, source <code class="docutils literal"><span class="pre">task_ids</span></code>, and source <code class="docutils literal"><span class="pre">dag_id</span></code>. By
+default, <code class="docutils literal"><span class="pre">xcom_pull()</span></code> filters for the keys that are automatically given to
+XComs when they are pushed by being returned from execute functions (as
+opposed to XComs that are pushed manually).</p>
+<p>If <code class="docutils literal"><span class="pre">xcom_pull</span></code> is passed a single string for <code class="docutils literal"><span class="pre">task_ids</span></code>, then the most
+recent XCom value from that task is returned; if a list of <code class="docutils literal"><span class="pre">task_ids</span></code> is
+passed, then a correpsonding list of XCom values is returned.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="c1"># inside a PythonOperator called 'pushing_task'</span>
+<span class="k">def</span> <span class="nf">push_function</span><span class="p">():</span>
+ <span class="k">return</span> <span class="n">value</span>
+
+<span class="c1"># inside another PythonOperator where provide_context=True</span>
+<span class="k">def</span> <span class="nf">pull_function</span><span class="p">(</span><span class="o">**</span><span class="n">context</span><span class="p">):</span>
+ <span class="n">value</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'task_instance'</span><span class="p">]</span><span class="o">.</span><span class="n">xcom_pull</span><span class="p">(</span><span class="n">task_ids</span><span class="o">=</span><span class="s1">'pushing_task'</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>It is also possible to pull XCom directly in a template, here’s an example
+of what this may look like:</p>
+<div class="code sql highlight-default"><div class="highlight"><pre><span></span><span class="n">SELECT</span> <span class="o">*</span> <span class="n">FROM</span> <span class="p">{{</span> <span class="n">task_instance</span><span class="o">.</span><span class="n">xcom_pull</span><span class="p">(</span><span class="n">task_ids</span><span class="o">=</span><span class="s1">'foo'</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="s1">'table_name'</span><span class="p">)</span> <span class="p">}}</span>
+</pre></div>
+</div>
+<p>Note that XComs are similar to <a class="reference internal" href="#variables">Variables</a>, but are specifically designed
+for inter-task communication rather than global settings.</p>
+</div>
+<div class="section" id="variables">
+<h3>Variables<a class="headerlink" href="#variables" title="Permalink to this headline">�</a></h3>
+<p>Variables are a generic way to store and retrieve arbitrary content or
+settings as a simple key value store within Airflow. Variables can be
+listed, created, updated and deleted from the UI (<code class="docutils literal"><span class="pre">Admin</span> <span class="pre">-></span> <span class="pre">Variables</span></code>),
+code or CLI. While your pipeline code definition and most of your constants
+and variables should be defined in code and stored in source control,
+it can be useful to have some variables or configuration items
+accessible and modifiable through the UI.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">Variable</span>
+<span class="n">foo</span> <span class="o">=</span> <span class="n">Variable</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"foo"</span><span class="p">)</span>
+<span class="n">bar</span> <span class="o">=</span> <span class="n">Variable</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">"bar"</span><span class="p">,</span> <span class="n">deserialize_json</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
+</pre></div>
+</div>
+<p>The second call assumes <code class="docutils literal"><span class="pre">json</span></code> content and will be deserialized into
+<code class="docutils literal"><span class="pre">bar</span></code>. Note that <code class="docutils literal"><span class="pre">Variable</span></code> is a sqlalchemy model and can be used
+as such.</p>
+</div>
+<div class="section" id="branching">
+<h3>Branching<a class="headerlink" href="#branching" title="Permalink to this headline">�</a></h3>
+<p>Sometimes you need a workflow to branch, or only go down a certain path
+based on an arbitrary condition which is typically related to something
+that happened in an upstream task. One way to do this is by using the
+<code class="docutils literal"><span class="pre">BranchPythonOperator</span></code>.</p>
+<p>The <code class="docutils literal"><span class="pre">BranchPythonOperator</span></code> is much like the PythonOperator except that it
+expects a python_callable that returns a task_id. The task_id returned
+is followed, and all of the other paths are skipped.
+The task_id returned by the Python function has to be referencing a task
+directly downstream from the BranchPythonOperator task.</p>
+<p>Note that using tasks with <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> downstream from
+<code class="docutils literal"><span class="pre">BranchPythonOperator</span></code> is logically unsound as <code class="docutils literal"><span class="pre">skipped</span></code> status
+will invariably lead to block tasks that depend on their past successes.
+<code class="docutils literal"><span class="pre">skipped</span></code> states propagates where all directly upstream tasks are
+<code class="docutils literal"><span class="pre">skipped</span></code>.</p>
+<p>If you want to skip some tasks, keep in mind that you can’t have an empty
+path, if so make a dummy task.</p>
+<p>like this, the dummy task “branch_false” is skipped</p>
+<img alt="_images/branch_good.png" src="_images/branch_good.png" />
+<p>Not like this, where the join task is skipped</p>
+<img alt="_images/branch_bad.png" src="_images/branch_bad.png" />
+</div>
+<div class="section" id="subdags">
+<h3>SubDAGs<a class="headerlink" href="#subdags" title="Permalink to this headline">�</a></h3>
+<p>SubDAGs are perfect for repeating patterns. Defining a function that returns a
+DAG object is a nice design pattern when using Airflow.</p>
+<p>Airbnb uses the <em>stage-check-exchange</em> pattern when loading data. Data is staged
+in a temporary table, after which data quality checks are performed against
+that table. Once the checks all pass the partition is moved into the production
+table.</p>
+<p>As another example, consider the following DAG:</p>
+<img alt="_images/subdag_before.png" src="_images/subdag_before.png" />
+<p>We can combine all of the parallel <code class="docutils literal"><span class="pre">task-*</span></code> operators into a single SubDAG,
+so that the resulting DAG resembles the following:</p>
+<img alt="_images/subdag_after.png" src="_images/subdag_after.png" />
+<p>Note that SubDAG operators should contain a factory method that returns a DAG
+object. This will prevent the SubDAG from being treated like a separate DAG in
+the main UI. For example:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="c1">#dags/subdag.py</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">DAG</span>
+<span class="kn">from</span> <span class="nn">airflow.operators</span> <span class="k">import</span> <span class="n">DummyOperator</span>
+
+
+<span class="c1"># Dag is returned by a factory method</span>
+<span class="k">def</span> <span class="nf">sub_dag</span><span class="p">(</span><span class="n">parent_dag_name</span><span class="p">,</span> <span class="n">child_dag_name</span><span class="p">,</span> <span class="n">start_date</span><span class="p">,</span> <span class="n">schedule_interval</span><span class="p">):</span>
+ <span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span>
+ <span class="s1">'</span><span class="si">%s</span><span class="s1">.</span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">parent_dag_name</span><span class="p">,</span> <span class="n">child_dag_name</span><span class="p">),</span>
+ <span class="n">schedule_interval</span><span class="o">=</span><span class="n">schedule_interval</span><span class="p">,</span>
+ <span class="n">start_date</span><span class="o">=</span><span class="n">start_date</span><span class="p">,</span>
+ <span class="p">)</span>
+
+ <span class="n">dummy_operator</span> <span class="o">=</span> <span class="n">DummyOperator</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="s1">'dummy_task'</span><span class="p">,</span>
+ <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">,</span>
+ <span class="p">)</span>
+
+ <span class="k">return</span> <span class="n">dag</span>
+</pre></div>
+</div>
+<p>This SubDAG can then be referenced in your main DAG file:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="c1"># main_dag.py</span>
+<span class="kn">from</span> <span class="nn">datetime</span> <span class="k">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timedelta</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">DAG</span>
+<span class="kn">from</span> <span class="nn">airflow.operators</span> <span class="k">import</span> <span class="n">SubDagOperator</span>
+<span class="kn">from</span> <span class="nn">dags.subdag</span> <span class="k">import</span> <span class="n">sub_dag</span>
+
+
+<span class="n">PARENT_DAG_NAME</span> <span class="o">=</span> <span class="s1">'parent_dag'</span>
+<span class="n">CHILD_DAG_NAME</span> <span class="o">=</span> <span class="s1">'child_dag'</span>
+
+<span class="n">main_dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span>
+ <span class="n">dag_id</span><span class="o">=</span><span class="n">PARENT_DAG_NAME</span><span class="p">,</span>
+ <span class="n">schedule_interval</span><span class="o">=</span><span class="n">timedelta</span><span class="p">(</span><span class="n">hours</span><span class="o">=</span><span class="mi">1</span><span class="p">),</span>
+ <span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2016</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
+<span class="p">)</span>
+
+<span class="n">sub_dag</span> <span class="o">=</span> <span class="n">SubDagOperator</span><span class="p">(</span>
+ <span class="n">subdag</span><span class="o">=</span><span class="n">sub_dag</span><span class="p">(</span><span class="n">PARENT_DAG_NAME</span><span class="p">,</span> <span class="n">CHILD_DAG_NAME</span><span class="p">,</span> <span class="n">main_dag</span><span class="o">.</span><span class="n">start_date</span><span class="p">,</span>
+ <span class="n">main_dag</span><span class="o">.</span><span class="n">schedule_interval</span><span class="p">),</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="n">CHILD_DAG_NAME</span><span class="p">,</span>
+ <span class="n">dag</span><span class="o">=</span><span class="n">main_dag</span><span class="p">,</span>
+<span class="p">)</span>
+</pre></div>
+</div>
+<p>You can zoom into a SubDagOperator from the graph view of the main DAG to show
+the tasks contained within the SubDAG:</p>
+<img alt="_images/subdag_zoom.png" src="_images/subdag_zoom.png" />
+<p>Some other tips when using SubDAGs:</p>
+<ul class="simple">
+<li>by convention, a SubDAG’s <code class="docutils literal"><span class="pre">dag_id</span></code> should be prefixed by its parent and
+a dot. As in <code class="docutils literal"><span class="pre">parent.child</span></code></li>
+<li>share arguments between the main DAG and the SubDAG by passing arguments to
+the SubDAG operator (as demonstrated above)</li>
+<li>SubDAGs must have a schedule and be enabled. If the SubDAG’s schedule is
+set to <code class="docutils literal"><span class="pre">None</span></code> or <code class="docutils literal"><span class="pre">@once</span></code>, the SubDAG will succeed without having done
+anything</li>
+<li>clearing a SubDagOperator also clears the state of the tasks within</li>
+<li>marking success on a SubDagOperator does not affect the state of the tasks
+within</li>
+<li>refrain from using <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> in tasks within the SubDAG as
+this can be confusing</li>
+<li>it is possible to specify an executor for the SubDAG. It is common to use
+the SequentialExecutor if you want to run the SubDAG in-process and
+effectively limit its parallelism to one. Using LocalExecutor can be
+problematic as it may over-subscribe your worker, running multiple tasks in
+a single slot</li>
+</ul>
+<p>See <code class="docutils literal"><span class="pre">airflow/example_dags</span></code> for a demonstration.</p>
+</div>
+<div class="section" id="slas">
+<h3>SLAs<a class="headerlink" href="#slas" title="Permalink to this headline">�</a></h3>
+<p>Service Level Agreements, or time by which a task or DAG should have
+succeeded, can be set at a task level as a <code class="docutils literal"><span class="pre">timedelta</span></code>. If
+one or many instances have not succeeded by that time, an alert email is sent
+detailing the list of tasks that missed their SLA. The event is also recorded
+in the database and made available in the web UI under <code class="docutils literal"><span class="pre">Browse->Missed</span> <span class="pre">SLAs</span></code>
+where events can be analyzed and documented.</p>
+</div>
+<div class="section" id="trigger-rules">
+<h3>Trigger Rules<a class="headerlink" href="#trigger-rules" title="Permalink to this headline">�</a></h3>
+<p>Though the normal workflow behavior is to trigger tasks when all their
+directly upstream tasks have succeeded, Airflow allows for more complex
+dependency settings.</p>
+<p>All operators have a <code class="docutils literal"><span class="pre">trigger_rule</span></code> argument which defines the rule by which
+the generated task get triggered. The default value for <code class="docutils literal"><span class="pre">trigger_rule</span></code> is
+<code class="docutils literal"><span class="pre">all_success</span></code> and can be defined as “trigger this task when all directly
+upstream tasks have succeeded”. All other rules described here are based
+on direct parent tasks and are values that can be passed to any operator
+while creating tasks:</p>
+<ul class="simple">
+<li><code class="docutils literal"><span class="pre">all_success</span></code>: (default) all parents have succeeded</li>
+<li><code class="docutils literal"><span class="pre">all_failed</span></code>: all parents are in a <code class="docutils literal"><span class="pre">failed</span></code> or <code class="docutils literal"><span class="pre">upstream_failed</span></code> state</li>
+<li><code class="docutils literal"><span class="pre">all_done</span></code>: all parents are done with their execution</li>
+<li><code class="docutils literal"><span class="pre">one_failed</span></code>: fires as soon as at least one parent has failed, it does not wait for all parents to be done</li>
+<li><code class="docutils literal"><span class="pre">one_success</span></code>: fires as soon as at least one parent succeeds, it does not wait for all parents to be done</li>
+<li><code class="docutils literal"><span class="pre">dummy</span></code>: dependencies are just for show, trigger at will</li>
+</ul>
+<p>Note that these can be used in conjunction with <code class="docutils literal"><span class="pre">depends_on_past</span></code> (boolean)
+that, when set to <code class="docutils literal"><span class="pre">True</span></code>, keeps a task from getting triggered if the
+previous schedule for the task hasn’t succeeded.</p>
+</div>
+<div class="section" id="zombies-undeads">
+<h3>Zombies & Undeads<a class="headerlink" href="#zombies-undeads" title="Permalink to this headline">�</a></h3>
+<p>Task instances die all the time, usually as part of their normal life cycle,
+but sometimes unexpectedly.</p>
+<p>Zombie tasks are characterized by the absence
+of an heartbeat (emitted by the job periodically) and a <code class="docutils literal"><span class="pre">running</span></code> status
+in the database. They can occur when a worker node can’t reach the database,
+when Airflow processes are killed externally, or when a node gets rebooted
+for instance. Zombie killing is performed periodically by the scheduler’s
+process.</p>
+<p>Undead processes are characterized by the existence of a process and a matching
+heartbeat, but Airflow isn’t aware of this task as <code class="docutils literal"><span class="pre">running</span></code> in the database.
+This mismatch typically occurs as the state of the database is altered,
+most likely by deleting rows in the “Task Instances” view in the UI.
+Tasks are instructed to verify their state as part of the heartbeat routine,
+and terminate themselves upon figuring out that they are in this “undead”
+state.</p>
+</div>
+<div class="section" id="cluster-policy">
+<h3>Cluster Policy<a class="headerlink" href="#cluster-policy" title="Permalink to this headline">�</a></h3>
+<p>Your local airflow settings file can define a <code class="docutils literal"><span class="pre">policy</span></code> function that
+has the ability to mutate task attributes based on other task or DAG
+attributes. It receives a single argument as a reference to task objects,
+and is expected to alter its attributes.</p>
+<p>For example, this function could apply a specific queue property when
+using a specific operator, or enforce a task timeout policy, making sure
+that no tasks run for more than 48 hours. Here’s an example of what this
+may look like inside your <code class="docutils literal"><span class="pre">airflow_settings.py</span></code>:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">policy</span><span class="p">(</span><span class="n">task</span><span class="p">):</span>
+ <span class="k">if</span> <span class="n">task</span><span class="o">.</span><span class="n">__class__</span><span class="o">.</span><span class="n">__name__</span> <span class="o">==</span> <span class="s1">'HivePartitionSensor'</span><span class="p">:</span>
+ <span class="n">task</span><span class="o">.</span><span class="n">queue</span> <span class="o">=</span> <span class="s2">"sensor_queue"</span>
+ <span class="k">if</span> <span class="n">task</span><span class="o">.</span><span class="n">timeout</span> <span class="o">></span> <span class="n">timedelta</span><span class="p">(</span><span class="n">hours</span><span class="o">=</span><span class="mi">48</span><span class="p">):</span>
+ <span class="n">task</span><span class="o">.</span><span class="n">timeout</span> <span class="o">=</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">hours</span><span class="o">=</span><span class="mi">48</span><span class="p">)</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="documentation-notes">
+<h3>Documentation & Notes<a class="headerlink" href="#documentation-notes" title="Permalink to this headline">�</a></h3>
+<p>It’s possible to add documentation or notes to your dags & task objects that
+become visible in the web interface (“Graph View” for dags, “Task Details” for
+tasks). There are a set of special task attributes that get rendered as rich
+content if defined:</p>
+<table border="1" class="docutils">
+<colgroup>
+<col width="38%" />
+<col width="62%" />
+</colgroup>
+<thead valign="bottom">
+<tr class="row-odd"><th class="head">attribute</th>
+<th class="head">rendered to</th>
+</tr>
+</thead>
+<tbody valign="top">
+<tr class="row-even"><td>doc</td>
+<td>monospace</td>
+</tr>
+<tr class="row-odd"><td>doc_json</td>
+<td>json</td>
+</tr>
+<tr class="row-even"><td>doc_yaml</td>
+<td>yaml</td>
+</tr>
+<tr class="row-odd"><td>doc_md</td>
+<td>markdown</td>
+</tr>
+<tr class="row-even"><td>doc_rst</td>
+<td>reStructuredText</td>
+</tr>
+</tbody>
+</table>
+<p>Please note that for dags, dag_md is the only attribute interpreted.</p>
+<p>This is especially useful if your tasks are built dynamically from
+configuration files, it allows you to expose the configuration that led
+to the related tasks in Airflow.</p>
+<p>This content will get rendered as markdown respectively in the “Graph View” and
+“Task Details” pages.</p>
+</div>
+<div class="section" id="jinja-templating">
+<h3>Jinja Templating<a class="headerlink" href="#jinja-templating" title="Permalink to this headline">�</a></h3>
+<p>Airflow leverages the power of
+<a class="reference external" href="http://jinja.pocoo.org/docs/dev/">Jinja Templating</a> and this can be a
+powerful tool to use in combination with macros (see the <a class="reference internal" href="code.html#macros"><span class="std std-ref">Macros</span></a> section).</p>
+<p>For example, say you want to pass the execution date as an environment variable
+to a Bash script using the <code class="docutils literal"><span class="pre">BashOperator</span></code>.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="c1"># The execution date as YYYY-MM-DD</span>
+<span class="n">date</span> <span class="o">=</span> <span class="s2">"{{ ds }}"</span>
+<span class="n">t</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span>
+ <span class="n">task_id</span><span class="o">=</span><span class="s1">'test_env'</span><span class="p">,</span>
+ <span class="n">bash_command</span><span class="o">=</span><span class="s1">'/tmp/test.sh '</span><span class="p">,</span>
+ <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">,</span>
+ <span class="n">env</span><span class="o">=</span><span class="p">{</span><span class="s1">'EXECUTION_DATE'</span><span class="p">:</span> <span class="n">date</span><span class="p">})</span>
+</pre></div>
+</div>
+<p>Here, <code class="docutils literal"><span class="pre">{{</span> <span class="pre">ds</span> <span class="pre">}}</span></code> is a macro, and because the <code class="docutils literal"><span class="pre">env</span></code> parameter of the
+<code class="docutils literal"><span class="pre">BashOperator</span></code> is templated with Jinja, the execution date will be available
+as an environment variable named <code class="docutils literal"><span class="pre">EXECUTION_DATE</span></code> in your Bash script.</p>
+<p>You can use Jinja templating with every parameter that is marked as “templated”
+in the documentation.</p>
+</div>
+</div>
+<div class="section" id="packaged-dags">
+<h2>Packaged dags<a class="headerlink" href="#packaged-dags" title="Permalink to this headline">�</a></h2>
+<p>While often you will specify dags in a single <code class="docutils literal"><span class="pre">.py</span></code> file it might sometimes
+be required to combine dag and its dependencies. For example, you might want
+to combine several dags together to version them together or you might want
+to manage them together or you might need an extra module that is not available
+by default on the system you are running airflow on. To allow this you can create
+a zip file that contains the dag(s) in the root of the zip file and have the extra
+modules unpacked in directories.</p>
+<p>For instance you can create a zip file that looks like this:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span>my_dag1.py
+my_dag2.py
+package1/__init__.py
+package1/functions.py
+</pre></div>
+</div>
+<p>Airflow will scan the zip file and try to load <code class="docutils literal"><span class="pre">my_dag1.py</span></code> and <code class="docutils literal"><span class="pre">my_dag2.py</span></code>.
+It will not go into subdirectories as these are considered to be potential
+packages.</p>
+<p>In case you would like to add module dependencies to your DAG you basically would
+do the same, but then it is more to use a virtualenv and pip.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span>virtualenv zip_dag
+<span class="nb">source</span> zip_dag/bin/activate
+
+mkdir zip_dag_contents
+<span class="nb">cd</span> zip_dag_contents
+
+pip install --install-option<span class="o">=</span><span class="s2">"--install-lib=</span><span class="nv">$PWD</span><span class="s2">"</span> my_useful_package
+cp ~/my_dag.py .
+
+zip -r zip_dag.zip *
+</pre></div>
+</div>
+<div class="admonition note">
+<p class="first admonition-title">Note</p>
+<p class="last">the zip file will be inserted at the beginning of module search list
+(sys.path) and as such it will be available to any other code that resides
+within the same interpreter.</p>
+</div>
+<div class="admonition note">
+<p class="first admonition-title">Note</p>
+<p class="last">packaged dags cannot be used with pickling turned on.</p>
+</div>
+<div class="admonition note">
+<p class="first admonition-title">Note</p>
+<p class="last">packaged dags cannot contain dynamic libraries (eg. libz.so) these need
+to be available on the system if a module needs those. In other words only
+pure python modules can be packaged.</p>
+</div>
+</div>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="profiling.html" class="btn btn-neutral float-right" title="Data Profiling" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="ui.html" class="btn btn-neutral" title="UI / Screenshots" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/configuration.html
----------------------------------------------------------------------
diff --git a/configuration.html b/configuration.html
new file mode 100644
index 0000000..3859b34
--- /dev/null
+++ b/configuration.html
@@ -0,0 +1,419 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Configuration — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="UI / Screenshots" href="ui.html"/>
+ <link rel="prev" title="Tutorial" href="tutorial.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">Configuration</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#setting-configuration-options">Setting Configuration Options</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#setting-up-a-backend">Setting up a Backend</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#connections">Connections</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#scaling-out-with-celery">Scaling Out with Celery</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#logs">Logs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#scaling-out-on-mesos-community-contributed">Scaling Out on Mesos (community contributed)</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#integration-with-systemd">Integration with systemd</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#integration-with-upstart">Integration with upstart</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>Configuration</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/configuration.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="configuration">
+<h1>Configuration<a class="headerlink" href="#configuration" title="Permalink to this headline">�</a></h1>
+<p>Setting up the sandbox in the <a class="reference internal" href="start.html"><span class="doc">Quick Start</span></a> section was easy;
+building a production-grade environment requires a bit more work!</p>
+<div class="section" id="setting-configuration-options">
+<h2>Setting Configuration Options<a class="headerlink" href="#setting-configuration-options" title="Permalink to this headline">�</a></h2>
+<p>The first time you run Airflow, it will create a file called <code class="docutils literal"><span class="pre">airflow.cfg</span></code> in
+your <code class="docutils literal"><span class="pre">$AIRFLOW_HOME</span></code> directory (<code class="docutils literal"><span class="pre">~/airflow</span></code> by default). This file contains Airflow’s configuration and you
+can edit it to change any of the settings. You can also set options with environment variables by using this format:
+<code class="docutils literal"><span class="pre">$AIRFLOW__{SECTION}__{KEY}</span></code> (note the double underscores).</p>
+<p>For example, the
+metadata database connection string can either be set in <code class="docutils literal"><span class="pre">airflow.cfg</span></code> like this:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>core<span class="o">]</span>
+<span class="nv">sql_alchemy_conn</span> <span class="o">=</span> my_conn_string
+</pre></div>
+</div>
+<p>or by creating a corresponding environment variable:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nv">AIRFLOW__CORE__SQL_ALCHEMY_CONN</span><span class="o">=</span>my_conn_string
+</pre></div>
+</div>
+<p>You can also derive the connection string at run time by appending <code class="docutils literal"><span class="pre">_cmd</span></code> to the key like this:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>core<span class="o">]</span>
+<span class="nv">sql_alchemy_conn_cmd</span> <span class="o">=</span> bash_command_to_run
+</pre></div>
+</div>
+<p>But only three such configuration elements namely sql_alchemy_conn, broker_url and celery_result_backend can be fetched as a command. The idea behind this is to not store passwords on boxes in plain text files. The order of precedence is as follows -</p>
+<ol class="arabic simple">
+<li>environment variable</li>
+<li>configuration in airflow.cfg</li>
+<li>command in airflow.cfg</li>
+<li>default</li>
+</ol>
+</div>
+<div class="section" id="setting-up-a-backend">
+<h2>Setting up a Backend<a class="headerlink" href="#setting-up-a-backend" title="Permalink to this headline">�</a></h2>
+<p>If you want to take a real test drive of Airflow, you should consider
+setting up a real database backend and switching to the LocalExecutor.</p>
+<p>As Airflow was built to interact with its metadata using the great SqlAlchemy
+library, you should be able to use any database backend supported as a
+SqlAlchemy backend. We recommend using <strong>MySQL</strong> or <strong>Postgres</strong>.</p>
+<div class="admonition note">
+<p class="first admonition-title">Note</p>
+<p class="last">If you decide to use <strong>Postgres</strong>, we recommend using the <code class="docutils literal"><span class="pre">psycopg2</span></code>
+driver and specifying it in your SqlAlchemy connection string.
+Also note that since SqlAlchemy does not expose a way to target a
+specific schema in the Postgres connection URI, you may
+want to set a default schema for your role with a
+command similar to <code class="docutils literal"><span class="pre">ALTER</span> <span class="pre">ROLE</span> <span class="pre">username</span> <span class="pre">SET</span> <span class="pre">search_path</span> <span class="pre">=</span> <span class="pre">airflow,</span> <span class="pre">foobar;</span></code></p>
+</div>
+<p>Once you’ve setup your database to host Airflow, you’ll need to alter the
+SqlAlchemy connection string located in your configuration file
+<code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code>. You should then also change the “executor”
+setting to use “LocalExecutor”, an executor that can parallelize task
+instances locally.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># initialize the database</span>
+airflow initdb
+</pre></div>
+</div>
+</div>
+<div class="section" id="connections">
+<h2>Connections<a class="headerlink" href="#connections" title="Permalink to this headline">�</a></h2>
+<p>Airflow needs to know how to connect to your environment. Information
+such as hostname, port, login and passwords to other systems and services is
+handled in the <code class="docutils literal"><span class="pre">Admin->Connection</span></code> section of the UI. The pipeline code you
+will author will reference the ‘conn_id’ of the Connection objects.</p>
+<img alt="_images/connections.png" src="_images/connections.png" />
+<p>By default, Airflow will save the passwords for the connection in plain text
+within the metadata database. The <code class="docutils literal"><span class="pre">crypto</span></code> package is highly recommended
+during installation. The <code class="docutils literal"><span class="pre">crypto</span></code> package does require that your operating
+system have libffi-dev installed.</p>
+<p>Connections in Airflow pipelines can be created using environment variables.
+The environment variable needs to have a prefix of <code class="docutils literal"><span class="pre">AIRFLOW_CONN_</span></code> for
+Airflow with the value in a URI format to use the connection properly. Please
+see the <a class="reference internal" href="concepts.html"><span class="doc">Concepts</span></a> documentation for more information on environment
+variables and connections.</p>
+</div>
+<div class="section" id="scaling-out-with-celery">
+<h2>Scaling Out with Celery<a class="headerlink" href="#scaling-out-with-celery" title="Permalink to this headline">�</a></h2>
+<p><code class="docutils literal"><span class="pre">CeleryExecutor</span></code> is one of the ways you can scale out the number of workers. For this
+to work, you need to setup a Celery backend (<strong>RabbitMQ</strong>, <strong>Redis</strong>, ...) and
+change your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> to point the executor parameter to
+<code class="docutils literal"><span class="pre">CeleryExecutor</span></code> and provide the related Celery settings.</p>
+<p>For more information about setting up a Celery broker, refer to the
+exhaustive <a class="reference external" href="http://docs.celeryproject.org/en/latest/getting-started/brokers/index.html">Celery documentation on the topic</a>.</p>
+<p>Here are a few imperative requirements for your workers:</p>
+<ul class="simple">
+<li><code class="docutils literal"><span class="pre">airflow</span></code> needs to be installed, and the CLI needs to be in the path</li>
+<li>Airflow configuration settings should be homogeneous across the cluster</li>
+<li>Operators that are executed on the worker need to have their dependencies
+met in that context. For example, if you use the <code class="docutils literal"><span class="pre">HiveOperator</span></code>,
+the hive CLI needs to be installed on that box, or if you use the
+<code class="docutils literal"><span class="pre">MySqlOperator</span></code>, the required Python library needs to be available in
+the <code class="docutils literal"><span class="pre">PYTHONPATH</span></code> somehow</li>
+<li>The worker needs to have access to its <code class="docutils literal"><span class="pre">DAGS_FOLDER</span></code>, and you need to
+synchronize the filesystems by your own means. A common setup would be to
+store your DAGS_FOLDER in a Git repository and sync it across machines using
+Chef, Puppet, Ansible, or whatever you use to configure machines in your
+environment. If all your boxes have a common mount point, having your
+pipelines files shared there should work as well</li>
+</ul>
+<p>To kick off a worker, you need to setup Airflow and kick off the worker
+subcommand</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span>airflow worker
+</pre></div>
+</div>
+<p>Your worker should start picking up tasks as soon as they get fired in
+its direction.</p>
+<p>Note that you can also run “Celery Flower”, a web UI built on top of Celery,
+to monitor your workers. You can use the shortcut command <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">flower</span></code>
+to start a Flower web server.</p>
+</div>
+<div class="section" id="logs">
+<h2>Logs<a class="headerlink" href="#logs" title="Permalink to this headline">�</a></h2>
+<p>Users can specify a logs folder in <code class="docutils literal"><span class="pre">airflow.cfg</span></code>. By default, it is in
+the <code class="docutils literal"><span class="pre">AIRFLOW_HOME</span></code> directory.</p>
+<p>In addition, users can supply a remote location for storing logs and log backups
+in cloud storage. At this time, Amazon S3 and Google Cloud Storage are supported.
+To enable this feature, <code class="docutils literal"><span class="pre">airflow.cfg</span></code> must be configured as in this example:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>core<span class="o">]</span>
+<span class="c1"># Airflow can store logs remotely in AWS S3 or Google Cloud Storage. Users</span>
+<span class="c1"># must supply a remote location URL (starting with either 's3://...' or</span>
+<span class="c1"># 'gs://...') and an Airflow connection id that provides access to the storage</span>
+<span class="c1"># location.</span>
+<span class="nv">remote_base_log_folder</span> <span class="o">=</span> s3://my-bucket/path/to/logs
+<span class="nv">remote_log_conn_id</span> <span class="o">=</span> MyS3Conn
+<span class="c1"># Use server-side encryption for logs stored in S3</span>
+<span class="nv">encrypt_s3_logs</span> <span class="o">=</span> False
+</pre></div>
+</div>
+<p>Remote logging uses an existing Airflow connection to read/write logs. If you don’t
+have a connection properly setup, this will fail. In the above example, Airflow will
+try to use <code class="docutils literal"><span class="pre">S3Hook('MyS3Conn')</span></code>.</p>
+<p>In the Airflow Web UI, local logs take precedance over remote logs. If local logs
+can not be found or accessed, the remote logs will be displayed. Note that logs
+are only sent to remote storage once a task completes (including failure). In other
+words, remote logs for running tasks are unavailable.</p>
+</div>
+<div class="section" id="scaling-out-on-mesos-community-contributed">
+<h2>Scaling Out on Mesos (community contributed)<a class="headerlink" href="#scaling-out-on-mesos-community-contributed" title="Permalink to this headline">�</a></h2>
+<p><code class="docutils literal"><span class="pre">MesosExecutor</span></code> allows you to schedule airflow tasks on a Mesos cluster.
+For this to work, you need a running mesos cluster and you must perform the following
+steps -</p>
+<ol class="arabic simple">
+<li>Install airflow on a machine where web server and scheduler will run,
+let’s refer to this as the “Airflow server”.</li>
+<li>On the Airflow server, install mesos python eggs from <a class="reference external" href="http://open.mesosphere.com/downloads/mesos/">mesos downloads</a>.</li>
+<li>On the Airflow server, use a database (such as mysql) which can be accessed from mesos
+slave machines and add configuration in <code class="docutils literal"><span class="pre">airflow.cfg</span></code>.</li>
+<li>Change your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> to point executor parameter to
+<cite>MesosExecutor</cite> and provide related Mesos settings.</li>
+<li>On all mesos slaves, install airflow. Copy the <code class="docutils literal"><span class="pre">airflow.cfg</span></code> from
+Airflow server (so that it uses same sql alchemy connection).</li>
+<li>On all mesos slaves, run the following for serving logs:</li>
+</ol>
+<div class="highlight-bash"><div class="highlight"><pre><span></span>airflow serve_logs
+</pre></div>
+</div>
+<ol class="arabic simple" start="7">
+<li>On Airflow server, to start processing/scheduling DAGs on mesos, run:</li>
+</ol>
+<div class="highlight-bash"><div class="highlight"><pre><span></span>airflow scheduler -p
+</pre></div>
+</div>
+<p>Note: We need -p parameter to pickle the DAGs.</p>
+<p>You can now see the airflow framework and corresponding tasks in mesos UI.
+The logs for airflow tasks can be seen in airflow UI as usual.</p>
+<p>For more information about mesos, refer to <a class="reference external" href="http://mesos.apache.org/documentation/latest/">mesos documentation</a>.
+For any queries/bugs on <cite>MesosExecutor</cite>, please contact <a class="reference external" href="https://github.com/kapil-malik">@kapil-malik</a>.</p>
+</div>
+<div class="section" id="integration-with-systemd">
+<h2>Integration with systemd<a class="headerlink" href="#integration-with-systemd" title="Permalink to this headline">�</a></h2>
+<p>Airflow can integrate with systemd based systems. This makes watching your
+daemons easy as systemd can take care of restarting a daemon on failure.
+In the <code class="docutils literal"><span class="pre">scripts/systemd</span></code> directory you can find unit files that
+have been tested on Redhat based systems. You can copy those to
+<code class="docutils literal"><span class="pre">/usr/lib/systemd/system</span></code>. It is assumed that Airflow will run under
+<code class="docutils literal"><span class="pre">airflow:airflow</span></code>. If not (or if you are running on a non Redhat
+based system) you probably need to adjust the unit files.</p>
+<p>Environment configuration is picked up from <code class="docutils literal"><span class="pre">/etc/sysconfig/airflow</span></code>.
+An example file is supplied. Make sure to specify the <code class="docutils literal"><span class="pre">SCHEDULER_RUNS</span></code>
+variable in this file when you run the scheduler. You
+can also define here, for example, <code class="docutils literal"><span class="pre">AIRFLOW_HOME</span></code> or <code class="docutils literal"><span class="pre">AIRFLOW_CONFIG</span></code>.</p>
+</div>
+<div class="section" id="integration-with-upstart">
+<h2>Integration with upstart<a class="headerlink" href="#integration-with-upstart" title="Permalink to this headline">�</a></h2>
+<p>Airflow can integrate with upstart based systems. Upstart automatically starts all airflow services for which you
+have a corresponding <code class="docutils literal"><span class="pre">*.conf</span></code> file in <code class="docutils literal"><span class="pre">/etc/init</span></code> upon system boot. On failure, upstart automatically restarts
+the process (until it reaches re-spawn limit set in a <code class="docutils literal"><span class="pre">*.conf</span></code> file).</p>
+<p>You can find sample upstart job files in the <code class="docutils literal"><span class="pre">scripts/upstart</span></code> directory. These files have been tested on
+Ubuntu 14.04 LTS. You may have to adjust <code class="docutils literal"><span class="pre">start</span> <span class="pre">on</span></code> and <code class="docutils literal"><span class="pre">stop</span> <span class="pre">on</span></code> stanzas to make it work on other upstart
+systems. Some of the possible options are listed in <code class="docutils literal"><span class="pre">scripts/upstart/README</span></code>.</p>
+<p>Modify <code class="docutils literal"><span class="pre">*.conf</span></code> files as needed and copy to <code class="docutils literal"><span class="pre">/etc/init</span></code> directory. It is assumed that airflow will run
+under <code class="docutils literal"><span class="pre">airflow:airflow</span></code>. Change <code class="docutils literal"><span class="pre">setuid</span></code> and <code class="docutils literal"><span class="pre">setgid</span></code> in <code class="docutils literal"><span class="pre">*.conf</span></code> files if you use other user/group</p>
+<p>You can use <code class="docutils literal"><span class="pre">initctl</span></code> to manually start, stop, view status of the airflow process that has been
+integrated with upstart</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span>initctl airflow-webserver status
+</pre></div>
+</div>
+</div>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="ui.html" class="btn btn-neutral float-right" title="UI / Screenshots" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="tutorial.html" class="btn btn-neutral" title="Tutorial" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[02/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/scheduler.html
----------------------------------------------------------------------
diff --git a/scheduler.html b/scheduler.html
new file mode 100644
index 0000000..4c703f8
--- /dev/null
+++ b/scheduler.html
@@ -0,0 +1,328 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Scheduling & Triggers — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="Plugins" href="plugins.html"/>
+ <link rel="prev" title="Command Line Interface" href="cli.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">Scheduling & Triggers</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#dag-runs">DAG Runs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#external-triggers">External Triggers</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#to-keep-in-mind">To Keep in Mind</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>Scheduling & Triggers</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/scheduler.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="scheduling-triggers">
+<h1>Scheduling & Triggers<a class="headerlink" href="#scheduling-triggers" title="Permalink to this headline">�</a></h1>
+<p>The Airflow scheduler monitors all tasks and all DAGs, and triggers the
+task instances whose dependencies have been met. Behind the scenes,
+it monitors and stays in sync with a folder for all DAG objects it may contain,
+and periodically (every minute or so) inspects active tasks to see whether
+they can be triggered.</p>
+<p>The Airflow scheduler is designed to run as a persistent service in an
+Airflow production environment. To kick it off, all you need to do is
+execute <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">scheduler</span></code>. It will use the configuration specified in
+<code class="docutils literal"><span class="pre">airflow.cfg</span></code>.</p>
+<p>Note that if you run a DAG on a <code class="docutils literal"><span class="pre">schedule_interval</span></code> of one day,
+the run stamped <code class="docutils literal"><span class="pre">2016-01-01</span></code> will be trigger soon after <code class="docutils literal"><span class="pre">2016-01-01T23:59</span></code>.
+In other words, the job instance is started once the period it covers
+has ended.</p>
+<p>The scheduler starts an instance of the executor specified in the your
+<code class="docutils literal"><span class="pre">airflow.cfg</span></code>. If it happens to be the <code class="docutils literal"><span class="pre">LocalExecutor</span></code>, tasks will be
+executed as subprocesses; in the case of <code class="docutils literal"><span class="pre">CeleryExecutor</span></code> and
+<code class="docutils literal"><span class="pre">MesosExecutor</span></code>, tasks are executed remotely.</p>
+<p>To start a scheduler, simply run the command:</p>
+<div class="code bash highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">scheduler</span>
+</pre></div>
+</div>
+<div class="section" id="dag-runs">
+<h2>DAG Runs<a class="headerlink" href="#dag-runs" title="Permalink to this headline">�</a></h2>
+<p>A DAG Run is an object representing an instantiation of the DAG in time.</p>
+<p>Each DAG may or may not have a schedule, which informs how <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Runs</span></code> are
+created. <code class="docutils literal"><span class="pre">schedule_interval</span></code> is defined as a DAG arguments, and receives
+preferably a
+<a class="reference external" href="https://en.wikipedia.org/wiki/Cron#CRON_expression">cron expression</a> as
+a <code class="docutils literal"><span class="pre">str</span></code>, or a <code class="docutils literal"><span class="pre">datetime.timedelta</span></code> object. Alternatively, you can also
+use one of these cron “preset”:</p>
+<table border="1" class="docutils">
+<colgroup>
+<col width="15%" />
+<col width="69%" />
+<col width="16%" />
+</colgroup>
+<thead valign="bottom">
+<tr class="row-odd"><th class="head">preset</th>
+<th class="head">Run once a year at midnight of January 1</th>
+<th class="head">cron</th>
+</tr>
+</thead>
+<tbody valign="top">
+<tr class="row-even"><td><code class="docutils literal"><span class="pre">None</span></code></td>
+<td>Don’t schedule, use for exclusively “externally triggered”
+DAGs</td>
+<td> </td>
+</tr>
+<tr class="row-odd"><td><code class="docutils literal"><span class="pre">@once</span></code></td>
+<td>Schedule once and only once</td>
+<td> </td>
+</tr>
+<tr class="row-even"><td><code class="docutils literal"><span class="pre">@hourly</span></code></td>
+<td>Run once an hour at the beginning of the hour</td>
+<td><code class="docutils literal"><span class="pre">0</span> <span class="pre">*</span> <span class="pre">*</span> <span class="pre">*</span> <span class="pre">*</span></code></td>
+</tr>
+<tr class="row-odd"><td><code class="docutils literal"><span class="pre">@daily</span></code></td>
+<td>Run once a day at midnight</td>
+<td><code class="docutils literal"><span class="pre">0</span> <span class="pre">0</span> <span class="pre">*</span> <span class="pre">*</span> <span class="pre">*</span></code></td>
+</tr>
+<tr class="row-even"><td><code class="docutils literal"><span class="pre">@weekly</span></code></td>
+<td>Run once a week at midnight on Sunday morning</td>
+<td><code class="docutils literal"><span class="pre">0</span> <span class="pre">0</span> <span class="pre">*</span> <span class="pre">*</span> <span class="pre">0</span></code></td>
+</tr>
+<tr class="row-odd"><td><code class="docutils literal"><span class="pre">@monthly</span></code></td>
+<td>Run once a month at midnight of the first day of the month</td>
+<td><code class="docutils literal"><span class="pre">0</span> <span class="pre">0</span> <span class="pre">1</span> <span class="pre">*</span> <span class="pre">*</span></code></td>
+</tr>
+<tr class="row-even"><td><code class="docutils literal"><span class="pre">@yearly</span></code></td>
+<td>Run once a year at midnight of January 1</td>
+<td><code class="docutils literal"><span class="pre">0</span> <span class="pre">0</span> <span class="pre">1</span> <span class="pre">1</span> <span class="pre">*</span></code></td>
+</tr>
+</tbody>
+</table>
+<p>Your DAG will be instantiated
+for each schedule, while creating a <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Run</span></code> entry for each schedule.</p>
+<p>DAG runs have a state associated to them (running, failed, success) and
+informs the scheduler on which set of schedules should be evaluated for
+task submissions. Without the metadata at the DAG run level, the Airflow
+scheduler would have much more work to do in order to figure out what tasks
+should be triggered and come to a crawl. It might also create undesired
+processing when changing the shape of your DAG, by say adding in new
+tasks.</p>
+</div>
+<div class="section" id="external-triggers">
+<h2>External Triggers<a class="headerlink" href="#external-triggers" title="Permalink to this headline">�</a></h2>
+<p>Note that <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Runs</span></code> can also be created manually through the CLI while
+running an <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">trigger_dag</span></code> command, where you can define a
+specific <code class="docutils literal"><span class="pre">run_id</span></code>. The <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Runs</span></code> created externally to the
+scheduler get associated to the trigger’s timestamp, and will be displayed
+in the UI alongside scheduled <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">runs</span></code>.</p>
+</div>
+<div class="section" id="to-keep-in-mind">
+<h2>To Keep in Mind<a class="headerlink" href="#to-keep-in-mind" title="Permalink to this headline">�</a></h2>
+<ul class="simple">
+<li>The first <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Run</span></code> is created based on the minimum <code class="docutils literal"><span class="pre">start_date</span></code> for the
+tasks in your DAG.</li>
+<li>Subsequent <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Runs</span></code> are created by the scheduler process, based on
+your DAG’s <code class="docutils literal"><span class="pre">schedule_interval</span></code>, sequentially.</li>
+<li>When clearing a set of tasks’ state in hope of getting them to re-run,
+it is important to keep in mind the <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Run</span></code>‘s state too as it defines
+whether the scheduler should look into triggering tasks for that run.</li>
+</ul>
+<p>Here are some of the ways you can <strong>unblock tasks</strong>:</p>
+<ul class="simple">
+<li>From the UI, you can <strong>clear</strong> (as in delete the status of) individual task instances from the task instances dialog, while defining whether you want to includes the past/future and the upstream/downstream dependencies. Note that a confirmation window comes next and allows you to see the set you are about to clear.</li>
+<li>The CLI command <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">clear</span> <span class="pre">-h</span></code> has lots of options when it comes to clearing task instance states, including specifying date ranges, targeting task_ids by specifying a regular expression, flags for including upstream and downstream relatives, and targeting task instances in specific states (<code class="docutils literal"><span class="pre">failed</span></code>, or <code class="docutils literal"><span class="pre">success</span></code>)</li>
+<li>Marking task instances as successful can be done through the UI. This is mostly to fix false negatives, or for instance when the fix has been applied outside of Airflow.</li>
+<li>The <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">backfill</span></code> CLI subcommand has a flag to <code class="docutils literal"><span class="pre">--mark_success</span></code> and allows selecting subsections of the DAG as well as specifying date ranges.</li>
+</ul>
+</div>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="plugins.html" class="btn btn-neutral float-right" title="Plugins" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="cli.html" class="btn btn-neutral" title="Command Line Interface" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/search.html
----------------------------------------------------------------------
diff --git a/search.html b/search.html
new file mode 100644
index 0000000..cdf8e86
--- /dev/null
+++ b/search.html
@@ -0,0 +1,214 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Search — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="#" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li></li>
+ <li class="wy-breadcrumbs-aside">
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <noscript>
+ <div id="fallback" class="admonition warning">
+ <p class="last">
+ Please activate JavaScript to enable the search
+ functionality.
+ </p>
+ </div>
+ </noscript>
+
+
+ <div id="search-results">
+
+ </div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+ <script type="text/javascript" src="_static/searchtools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+ <script type="text/javascript">
+ jQuery(function() { Search.loadIndex("searchindex.js"); });
+ </script>
+
+ <script type="text/javascript" id="searchindexloader"></script>
+
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/searchindex.js
----------------------------------------------------------------------
diff --git a/searchindex.js b/searchindex.js
new file mode 100644
index 0000000..434ea1e
--- /dev/null
+++ b/searchindex.js
@@ -0,0 +1 @@
+Search.setIndex({envversion:47,filenames:["cli","code","concepts","configuration","faq","index","installation","license","plugins","profiling","project","scheduler","security","start","tutorial","ui"],objects:{"airflow.contrib":{executors:[1,0,0,"-"],hooks:[1,0,0,"-"],operators:[1,0,0,"-"]},"airflow.contrib.hooks":{CloudantHook:[1,1,1,""],FTPHook:[1,1,1,""],SSHHook:[1,1,1,""],VerticaHook:[1,1,1,""]},"airflow.contrib.hooks.CloudantHook":{db:[1,2,1,""]},"airflow.contrib.hooks.FTPHook":{close_conn:[1,2,1,""],create_directory:[1,2,1,""],delete_directory:[1,2,1,""],delete_file:[1,2,1,""],describe_directory:[1,2,1,""],get_conn:[1,2,1,""],list_directory:[1,2,1,""],retrieve_file:[1,2,1,""],store_file:[1,2,1,""]},"airflow.contrib.hooks.SSHHook":{Popen:[1,2,1,""],check_output:[1,2,1,""],tunnel:[1,2,1,""]},"airflow.contrib.hooks.VerticaHook":{get_conn:[1,2,1,""]},"airflow.contrib.operators":{SSHExecuteOperator:[1,1,1,""],VerticaOperator:[1,1,1,""],VerticaToHiveTransfer:[1,1,1,""]},"airflow.con
trib.operators.hipchat_operator":{HipChatAPIOperator:[1,1,1,""],HipChatAPISendRoomNotificationOperator:[1,1,1,""]},"airflow.executors":{CeleryExecutor:[1,1,1,""],LocalExecutor:[1,1,1,""],SequentialExecutor:[1,1,1,""]},"airflow.hooks":{DbApiHook:[1,1,1,""],DruidHook:[1,1,1,""],HiveCliHook:[1,1,1,""],HiveMetastoreHook:[1,1,1,""],HiveServer2Hook:[1,1,1,""],HttpHook:[1,1,1,""],MsSqlHook:[1,1,1,""],MySqlHook:[1,1,1,""],PostgresHook:[1,1,1,""],PrestoHook:[1,1,1,""],S3Hook:[1,1,1,""],SqliteHook:[1,1,1,""],WebHDFSHook:[1,1,1,""]},"airflow.hooks.DbApiHook":{bulk_dump:[1,2,1,""],bulk_load:[1,2,1,""],get_conn:[1,2,1,""],get_cursor:[1,2,1,""],get_first:[1,2,1,""],get_pandas_df:[1,2,1,""],get_records:[1,2,1,""],insert_rows:[1,2,1,""],run:[1,2,1,""]},"airflow.hooks.DruidHook":{construct_ingest_query:[1,2,1,""],get_conn:[1,2,1,""],load_from_hdfs:[1,2,1,""]},"airflow.hooks.HiveCliHook":{load_file:[1,2,1,""],run_cli:[1,2,1,""],test_hql:[1,2,1,""]},"airflow.hooks.HiveMetastoreHook":{check_for_partiti
on:[1,2,1,""],get_databases:[1,2,1,""],get_metastore_client:[1,2,1,""],get_partitions:[1,2,1,""],get_table:[1,2,1,""],get_tables:[1,2,1,""],max_partition:[1,2,1,""],table_exists:[1,2,1,""]},"airflow.hooks.HiveServer2Hook":{get_pandas_df:[1,2,1,""],get_records:[1,2,1,""]},"airflow.hooks.HttpHook":{get_conn:[1,2,1,""],run:[1,2,1,""],run_and_check:[1,2,1,""]},"airflow.hooks.MsSqlHook":{get_conn:[1,2,1,""]},"airflow.hooks.MySqlHook":{bulk_load:[1,2,1,""],get_conn:[1,2,1,""]},"airflow.hooks.PrestoHook":{get_conn:[1,2,1,""],get_first:[1,2,1,""],get_pandas_df:[1,2,1,""],get_records:[1,2,1,""],run:[1,2,1,""]},"airflow.hooks.S3Hook":{check_for_bucket:[1,2,1,""],check_for_key:[1,2,1,""],check_for_prefix:[1,2,1,""],check_for_wildcard_key:[1,2,1,""],get_bucket:[1,2,1,""],get_conn:[1,2,1,""],get_key:[1,2,1,""],get_wildcard_key:[1,2,1,""],list_keys:[1,2,1,""],list_prefixes:[1,2,1,""],load_file:[1,2,1,""],load_string:[1,2,1,""]},"airflow.hooks.SqliteHook":{get_conn:[1,2,1,""]},"airflow.hooks.WebHD
FSHook":{check_for_path:[1,2,1,""],get_conn:[1,2,1,""],load_file:[1,2,1,""]},"airflow.macros":{ds_add:[1,3,1,""],ds_format:[1,3,1,""],hive:[1,0,0,"-"],integrate_plugins:[1,3,1,""],random:[1,3,1,""]},"airflow.macros.hive":{closest_ds_partition:[1,3,1,""],max_partition:[1,3,1,""]},"airflow.models":{BaseOperator:[1,1,1,""],Connection:[1,1,1,""],DAG:[1,1,1,""],DagBag:[1,1,1,""],TaskInstance:[1,1,1,""]},"airflow.models.BaseOperator":{clear:[1,2,1,""],dag:[1,4,1,""],detect_downstream_cycle:[1,2,1,""],downstream_list:[1,4,1,""],execute:[1,2,1,""],get_direct_relatives:[1,2,1,""],get_flat_relatives:[1,2,1,""],get_task_instances:[1,2,1,""],has_dag:[1,2,1,""],on_kill:[1,2,1,""],post_execute:[1,2,1,""],pre_execute:[1,2,1,""],prepare_template:[1,2,1,""],render_template:[1,2,1,""],render_template_from_field:[1,2,1,""],run:[1,2,1,""],schedule_interval:[1,4,1,""],set_downstream:[1,2,1,""],set_upstream:[1,2,1,""],upstream_list:[1,4,1,""],xcom_pull:[1,2,1,""],xcom_push:[1,2,1,""]},"airflow.models.Con
nection":{extra_dejson:[1,4,1,""]},"airflow.models.DAG":{add_task:[1,2,1,""],add_tasks:[1,2,1,""],cli:[1,2,1,""],concurrency_reached:[1,4,1,""],crawl_for_tasks:[1,2,1,""],create_dagrun:[1,2,1,""],filepath:[1,4,1,""],folder:[1,4,1,""],get_template_env:[1,2,1,""],is_paused:[1,4,1,""],latest_execution_date:[1,4,1,""],run:[1,2,1,""],set_dependency:[1,2,1,""],sub_dag:[1,2,1,""],subdags:[1,4,1,""],tree_view:[1,2,1,""]},"airflow.models.DagBag":{bag_dag:[1,2,1,""],collect_dags:[1,2,1,""],dagbag_report:[1,2,1,""],get_dag:[1,2,1,""],kill_zombies:[1,2,1,""],process_file:[1,2,1,""],size:[1,2,1,""]},"airflow.models.TaskInstance":{are_dependencies_met:[1,2,1,""],are_dependents_done:[1,2,1,""],clear_xcom_data:[1,2,1,""],command:[1,2,1,""],current_state:[1,2,1,""],error:[1,2,1,""],evaluate_trigger_rule:[1,2,1,""],is_premature:[1,2,1,""],is_queueable:[1,2,1,""],is_runnable:[1,2,1,""],key:[1,4,1,""],pool_full:[1,2,1,""],ready_for_retry:[1,2,1,""],refresh_from_db:[1,2,1,""],run:[1,2,1,""],xcom_pull:[1
,2,1,""],xcom_push:[1,2,1,""]},"airflow.operators":{BashOperator:[1,1,1,""],BranchPythonOperator:[1,1,1,""],DummyOperator:[1,1,1,""],EmailOperator:[1,1,1,""],ExternalTaskSensor:[1,1,1,""],GenericTransfer:[1,1,1,""],HdfsSensor:[1,1,1,""],Hive2SambaOperator:[1,1,1,""],HiveOperator:[1,1,1,""],HivePartitionSensor:[1,1,1,""],HiveToDruidTransfer:[1,1,1,""],HiveToMySqlTransfer:[1,1,1,""],HttpSensor:[1,1,1,""],MetastorePartitionSensor:[1,1,1,""],MsSqlOperator:[1,1,1,""],MsSqlToHiveTransfer:[1,1,1,""],MySqlOperator:[1,1,1,""],MySqlToHiveTransfer:[1,1,1,""],PostgresOperator:[1,1,1,""],PrestoCheckOperator:[1,1,1,""],PrestoIntervalCheckOperator:[1,1,1,""],PrestoValueCheckOperator:[1,1,1,""],PythonOperator:[1,1,1,""],S3KeySensor:[1,1,1,""],S3ToHiveTransfer:[1,1,1,""],ShortCircuitOperator:[1,1,1,""],SimpleHttpOperator:[1,1,1,""],SlackAPIOperator:[1,1,1,""],SlackAPIPostOperator:[1,1,1,""],SqlSensor:[1,1,1,""],TimeSensor:[1,1,1,""],TriggerDagRunOperator:[1,1,1,""],WebHdfsSensor:[1,1,1,""]},"airflow
.operators.BashOperator":{execute:[1,2,1,""]},"airflow.operators.SlackAPIOperator":{construct_api_call_params:[1,2,1,""],execute:[1,2,1,""]},"airflow.operators.docker_operator":{DockerOperator:[1,1,1,""]},"airflow.operators.sensors":{BaseSensorOperator:[1,1,1,""]},airflow:{executors:[1,0,0,"-"],hooks:[1,0,0,"-"],macros:[1,0,0,"-"],models:[1,0,0,"-"],operators:[1,0,0,"-"]}},objnames:{"0":["py","module","Python module"],"1":["py","class","Python class"],"2":["py","method","Python method"],"3":["py","function","Python function"],"4":["py","attribute","Python attribute"]},objtypes:{"0":"py:module","1":"py:class","2":"py:method","3":"py:function","4":"py:attribute"},terms:{"00am":1,"01t23":11,"10gb":1,"10pm":2,"128m":1,"2am":1,"5gb":1,"9ba53fba14eb":[],"__class__":2,"__init__":[1,2],"__name__":[2,8],"_cmd":3,"_host":12,"abstract":[1,2,5,8],"boolean":[1,2],"byte":[1,4],"case":[1,2,4,11,12,14],"class":[0,1,2,6,7,8,14],"default":[],"export":[1,13],"final":1,"float":1,"function":[],"import":
[],"int":1,"long":1,"new":[1,2,4,8,11,12],"return":[0,1,2,8],"short":1,"static":[1,5,8],"super":[1,12],"switch":[3,12],"true":[1,2,4,12,14],"try":[1,2,3],"var":1,"while":[1,2,4,5,7,11,13,14,15],abbrevi:2,abil:2,abl:[3,4,12,13,14],about:[1,2,3,7,10,11,14],abov:[2,3,7,12,15],absenc:2,absolut:2,accept:[1,7],access:[1,2,3,8,12],access_token:15,accord:12,accordingli:2,account:[1,12],acquir:1,across:[3,15],act:[1,2,7,14],action:[1,15],activ:[1,2,4,8,11,12,14],actual:[1,2,4,14],acycl:[1,2,5],add:[1,2,3,7,12],add_task:1,addendum:7,addit:[],addition:12,addprinc:12,adhoc:[],adjust:[3,12],admin:[2,3,4,8,9,12,13],admin_view:8,advanc:14,advantag:12,advis:[1,7],affect:2,after:[1,2,4,11],afterward:1,against:[1,2,4,7,12,14],agari:10,aggregate_db_message_job:2,aggregate_db_message_job_cmd:2,ago:4,agre:7,agreement:[2,7],airbnb:[1,2,10,14],airflow:[],airflow__:3,airflow__core__sql_alchemy_conn:3,airflow_config:3,airflow_conn_:[2,3],airflow_conn_postgres_mast:2,airflow_hom:[3,4,8,13],airflow_set:2,airf
low_tmp_dir:1,airflowignor:1,airflowplugin:8,airflowtestplugin:8,aiyzngq:1,alchemi:3,alert:[1,2,8],all:[0,1,2,3,4,6,7,11,12,14],all_db:6,all_don:[1,2],all_fail:[1,2],all_success:[1,2],alleg:7,alloc:1,allow:[0,1,2,3,4,5,8,9,11,12,13,14,15],allowed_st:1,allowed_team:12,alon:7,along:[1,4,7,14],alongsid:[1,7,11],alreadi:1,alright:14,also:[1,2,3,4,6,7,8,11,12,14],alter:[1,2,3,4,12],altern:[1,11],altert:1,although:12,alwai:[1,2,15],amazon:3,amount:[1,2],analys:15,analyz:2,anand:10,anchor:1,ani:[1,2,3,4,7,14,15],annot:7,announc:10,anomali:8,anoth:[1,2,4,10],ansibl:3,anyon:12,anyth:[2,14],anytim:2,anywher:[1,2],aoen:10,apach:[],api:[],api_call_param:1,api_kei:15,api_param:1,api_vers:1,apikei:15,apiv2:1,appar:1,appear:[1,2,7],append:3,appendix:7,appli:[1,2,6,7,11],applic:[7,8,9,12],appropri:[2,7,12],arbitrari:[2,5],architectur:5,archiv:7,are_dependencies_met:1,are_dependents_don:1,aren:[1,2],arg:1,argument:[],aris:7,around:[1,5,14],arrai:[1,5,8],arthur:10,artwr:10,ascii:1,assert:7,asset:1,as
sign:[],associ:[1,2,7,11],assum:[1,2,3,7,14],asterisk:12,async:6,atom:2,attach:[1,2,7],attach_to:1,attack:12,attempt:[0,1],attent:[1,4],attr:1,attribut:[0,1,2,7],audit:8,auth:[1,12],auth_backend:12,authen:12,authent:[],authmechan:1,author:[1,3,5,7,12,14,15],authorship:7,autocommit:1,automat:[1,2,3],avail:[1,2,3,7,8,9],averag:1,avoid:2,awar:2,azkaban:5,back:[1,4,8],backend:[],backfil:[],background:[1,14],backup:3,bag:1,bag_dag:1,bahchi:[],balanc:2,bar:[1,2],bare:14,base64:4,base:[1,2,3,4,7,8,11],base_executor:[1,8],base_hook:[1,8],base_url:1,basedn:12,baseexecutor:[1,8],basehook:[1,2,8],baseview:8,bash:[1,2,14],bash_1:2,bash_command:[1,2,14],bash_command_to_run:3,bash_queu:14,bashoper:[1,2,14],basi:7,basic:[2,6,8,9,14],bcrypt:12,beauchemin:10,becaus:2,becom:[1,2,5,8,14],beelin:1,been:[1,2,3,7,11,12],befor:[0,1,2,4,12],begin:[2,11],behalf:7,behav:1,behavior:[1,2],behind:[1,3,6,11],belong:12,below:[1,7,8,13,14],benefici:7,best:[1,10],better:14,between:[1,2,14],beyond:[],bin:2,bind:7,bi
nd_password:12,bind_us:12,bit:3,blob:14,block:[1,2,14,15],blog:[7,10],blueprint:8,bodi:1,boilerpl:7,bolk:10,bolkedebruin:10,bonu:1,bool:1,boot:3,both:[1,4],bother:14,boto:1,bottleneck:15,box:[1,3,13],bracket:[7,14],branch:[],branch_fals:2,branchpythonoper:[1,2],broker:[0,3],broker_api:0,broker_url:3,brought:10,brows:[2,4],bruin:10,bucket:[1,3],bucket_kei:1,bucket_nam:1,buffer:1,bug:3,build:[],built:[1,2,3,5,8,9,14],bulk:[4,15],bulk_dump:1,bulk_load:1,bump:2,bundl:1,burn:0,but_this_dag_will_not:2,c12318391:1,cacert:12,cach:12,call:[1,2,3,14],callabl:1,callback:12,can:[1,2,3,4,5,6,8,9,11,12,13,14,15],cannot:[1,2,7,14],capabl:8,capac:2,card:1,care:[0,3],carri:[2,7],cast:1,cat:1,categori:8,caus:[4,7],celeri:[],celery_result_backend:3,celeryexecutor:[1,2,3,4,6,11],cell:1,central:2,cert:1,certain:[1,2],certif:1,cfg:[1,2,3,4,11,12,13,14],chain:2,chang:[3,5,7,11,12,13,14],channel:[1,10],charact:7,character:2,charg:7,charset:1,chart:[],chat:10,check:[0,1,2,4],check_for_bucket:1,check_for_kei
:1,check_for_partit:1,check_for_path:1,check_for_prefix:1,check_for_wildcard_kei:1,check_head:1,check_oper:1,check_output:1,checkoper:1,chef:[3,10],child:[1,2],child_dag:2,child_dag_nam:2,chmod:12,choic:[0,1,14],choos:[1,2,7],chri:10,circuit:1,claim:7,clariti:5,claus:1,clean:1,cleanup:1,clear:[0,1,2,4,11,15],clear_xcom_data:1,cli:[1,2,3,4,11],click:[4,12,14,15],client:[1,6,12],client_id:12,client_secret:12,close:[1,4,12],close_conn:1,closest:1,closest_ds_partit:1,cloud:[3,6],cloudant:[1,6],cloudant_conn_id:1,cloudant_default:1,cloudanthook:1,cmd:1,code:[],col:1,collabor:5,collect:[1,2,8],collect_dag:1,color:1,column:1,com:[1,12,14],combin:[2,7],come:[1,11],comma:[0,1,2],comment:7,commerci:7,commit:[1,10,12],commit_everi:1,common:[2,3,4,7,10,14],commonli:1,compani:8,compar:5,compil:[4,7],complet:[1,2,3],complex:[1,2,5,8],compli:7,complianc:7,complic:14,compon:8,compos:[2,12,14],comprehens:15,comput:7,conceptu:2,concern:2,concur:4,concurr:[0,1,4],concurrency_reach:1,condit:[1,2,6,7],c
onf:[0,1,3],config:[8,12],configur:[],confirm:[0,4,11],confluenc:[],confus:[1,2,4],conjunct:[0,2,13],conn_id:[1,2,3],conn_typ:1,connect:[],connect_timeout:1,connecton:9,consequenti:7,consid:[1,2,3],consider:1,consist:[2,7],conspicu:7,constant:[1,2],constitu:2,constitut:7,constraint:1,constru:7,construct:1,construct_api_call_param:1,construct_ingest_queri:1,constructor:[1,14],contact:[3,7],contain:[1,2,3,7,11,12,14,15],content:[],context:[],continu:[1,5],contract:7,contrib:[1,2,12],contributor:[7,10],contributori:7,control:[1,2,7,15],conveni:[1,2],convent:2,convers:7,cookbook:[],cool:1,coordin:1,copi:[1,3,7,12],copyright:[7,12],core:[],correct:2,correpsond:2,correspond:[1,2,3,4],could:[1,2,14],count:1,counterclaim:7,cours:12,cover:11,cpu:[1,8],craft:1,crawl:[1,11],crawl_for_task:1,creat:[1,2,3,4,8,9,11,12,13,14,15],create_dagrun:1,create_directori:1,creation:1,credenti:12,credit:12,criccomini:10,criteria:[1,2],criterion:1,critic:[1,2],cron:[1,4,11],cross:[2,7,14],crt:12,crypto:[3,4,6
],csv:1,curli:14,current:[1,15],current_st:1,cursor:1,custom:8,customari:7,cwiki:[],cycl:[1,2,14],daemon:[0,1,3],dag:[],dag_1:2,dag_2:2,dag_fold:[1,2],dag_id:[0,1,2,14],dag_md:2,dag_run:1,dag_run_obj:1,dagbag:1,dagbag_report:1,dagfil:2,dagrun:[0,1,4],dagrun_timeout:1,dags_fold:3,dai:[1,11,13,14],daili:[1,4,11],damag:7,dan:[1,10],data:[],data_profiler_filt:12,databas:[0,1,2,3,5,6,8,9,13,14],datafram:1,dataset:1,datasourc:1,date:[0,1,2,7,11,14],date_filter_column:1,datestamp:1,datetim:[1,2,4,11,14],dateutil:1,davydov:10,days_back:1,dbapi_hook:1,dbapihook:1,deadlock:[],deal:4,debug:[0,1,14],decid:[1,3],declar:1,deep:1,def:[1,2,8],default_arg:[1,2,4,14],default_login:12,default_queu:2,defend:7,defer:2,deferred_op:2,defin:[1,2,3,4,5,7,8,11,12,14],definit:[],del:1,delai:[1,4],delegate_to:[],delet:[1,2,11,15],delete_directori:1,delete_fil:1,deliber:[1,7],delimit:[0,1,2],demonstr:2,depend:[],depends_on_past:[0,1,2,4,14],deploi:8,deploy:8,depth:14,deriv:[1,3,7,8],descend:2,describ:[1,2,7],de
scribe_directori:1,descript:[1,2,7],deseri:[0,1,2],deserialize_json:2,design:[2,7,11],desir:[1,14],destin:1,destination_conn_id:1,destination_filepath:1,destination_t:1,detail:[1,2,4,12,15],detect:[1,8],detect_downstream_cycl:1,determin:[1,2,7],dev:[3,6,10],devel:6,devel_hadoop:6,develop:[0,1,12,14],deviat:1,dgaoqozlvea:[],dialog:11,dict:[0,1,2],dictionari:[1,2,14],did:14,die:2,differ:[1,2,7,8,14,15],dimens:1,direct:[1,2,3,5,7],directli:[1,2,4,9],directori:[0,1,2,3,12,14],dirti:13,disabl:1,disclaim:7,discuss:7,disk:1,displai:[3,7,11,14],disregard:[1,14],distinct:1,distribut:[1,2,6,7],do_pickl:0,doc:[1,2,4,14],doc_json:2,doc_md:2,doc_rst:2,doc_yaml:2,docker:[1,10],docker_oper:1,docker_url:1,dockeroper:[1,2],document:[],doe:[0,1,2,3,4,6,7,12,14],does_not_exist:1,doesn:[1,2,14],domain:12,don:[1,2,3,6,7,11],done:[1,2,11,12,14],donot_pickl:[0,1],door:2,dot:[1,2],doubl:[1,3,14],down:[0,2],download:[1,3],downstream:[0,1,2,11],downstream_list:1,downstream_task_id:1,drive:3,driven:8,driver:3
,drop:[1,8],druid:[1,6],druid_datasourc:1,druid_ingest_conn_id:1,druid_ingest_default:1,druid_query_conn_id:1,druid_query_default:1,druidhook:1,dry:0,dry_run:0,ds_add:[1,14],ds_format:1,ds_nodash:1,dubiou:1,dummi:[1,2,8],dummy_1:2,dummy_oper:2,dummy_task:2,dummyoper:[1,2],dump:1,durat:[],dure:[3,15],dynam:[1,2,4,5],each:[1,2,4,7,10,11,14],earlier:14,easi:[2,3,5,8,9,12,15],easier:[1,7],easiest:6,easili:[1,2,5,9,14],echo:[2,14],ecosystem:8,edit:[3,15],editori:7,effect:2,effici:1,egg:3,either:[1,2,3,7,12,13],elabor:7,elaps:1,electron:7,eleg:5,element:3,els:13,elsewher:2,email:[1,2,10,12,14],email_on_failur:[1,14],email_on_retri:[1,14],emailoper:[1,2],emit:[2,14],empti:[1,2],enabl:[],enclos:7,encod:[1,4],encrypt:[1,3,4,6,12],encrypt_s3_log:3,end:[1,7,11],end_dat:[0,1,14],endpoint:1,enforc:[2,4],engin:[1,4,5],enterpris:[],entir:[0,4],entiti:7,entri:[1,11],env:[1,2],environ:[1,2,3,5,6,9,11,14,15],ep_data_pipeline_db_msg_agg:2,equival:[2,6,14],era:4,error:1,especi:[2,4,14],essenti:1,etc:[1
,2,3,12],eval:1,evalu:[1,2,4,11,14],evaluate_trigger_rul:1,even:[1,2,7,9],event:[2,7,14],eventlet:0,ever:1,everi:[1,2,11,12,14],everyon:14,everyth:[14,15],exactli:[1,15],exampl:[],example1:13,example_bash_oper:13,example_dag:[2,14],example_team_1:12,example_team_2:12,exce:1,except:[1,2,4,7,14],exchang:[2,5],exclud:7,exclus:[7,11],execut:[0,1,2,3,4,5,7,11,14],execution_d:[0,1,2,14],execution_delta:1,execution_timeout:[1,2],exercis:7,exhaust:3,exist:[0,1,2,3,4,14],exit:[0,12],expect:[1,2,4,5],expir:1,explain:1,explan:14,explicit:5,explicit_op:2,explicitli:[1,2,7,12,14],expos:[1,2,3,8,12],express:[1,4,7,11],ext:1,extend:5,extens:[1,5],extern:[],external_dag_id:1,external_task_id:1,external_trigg:1,externaltasksensor:1,extra:[],extra_dejson:1,extra_opt:1,fact:2,factori:2,fail:[0,1,2,3,4,11,14,15],failur:[1,3,4,7],fairli:13,fall:10,fals:[0,1,3,11,14],familiar:[2,14],faq:[],fast:13,fault:2,featur:[1,2,3,6,8,14,15],feb:12,fee:7,fernet:4,fernet_kei:4,fetch:3,few:[1,3,9,13,14],field:[1,7],fi
eld_dict:1,fifti:7,figur:[1,2,11],file:[],filenam:[1,2],filepath:1,filestatu:1,filesystem:3,fill:[1,2,9,12],filter:[0,1,2,12],filter_by_own:12,find:[1,3,4,14,15],fine:14,finish:1,fire:[1,2,3],first:[0,1,2,3,4,10,11,13,14],fit:[1,5,7],fix:[1,11],flag:[1,11],flag_upstream_fail:1,flask:[0,8,9],flask_admin:8,flask_blueprint:8,flask_login:12,flat:1,flavor:8,flexibl:[1,14],flower:[0,3],folder:[1,3,8,11,13,14],follow:[1,2,3,5,7,12,14],foo:[1,2],foo_dag_id:4,foobar:3,forc:[0,1],force_pul:1,foreground:0,foreign:1,form:[],format:[1,2,3,7],forward:[1,4],found:[1,3,12],foundat:[7,10,13],four:2,framework:[3,8],free:[2,7],frm:1,from:[0,1,2,3,4,5,7,8,9,10,11,12,13,14,15],ftp:1,ftp_conn_id:1,ftp_default:1,ftphook:1,full:[1,10],full_filepath:1,fulli:[4,12],functiontyp:1,fundament:14,further:[13,14],futur:[1,11],galaxi:8,gantt:[],garag:2,gate:12,gc_base_hook:[],gcloud:[],gcp_api:6,gender:1,gener:[0,1,2,5,7,8,12,14,15],generictransf:1,get:[],get_active_run:[],get_bucket:1,get_conn:1,get_connect:2,get_
cursor:1,get_dag:1,get_databas:1,get_direct_rel:1,get_first:1,get_flat_rel:1,get_kei:1,get_metastore_cli:1,get_pandas_df:1,get_partit:1,get_partitions_by_filt:1,get_record:1,get_tabl:1,get_task_inst:1,get_template_context:1,get_template_env:1,get_wildcard_kei:1,gevent:0,ghe:[],ghe_oauth:12,ghost:1,git:3,github:[],github_enterpris:12,github_enterprise_auth:12,githubusercont:1,gitignor:1,gitter:10,give:[2,7],given:[1,2],glanc:15,global:[1,2,4],goal:14,goe:[1,4,14],good:2,goodby:2,goodwil:7,googl:[3,6],google_cloud_storage_conn_id:[],google_cloud_storage_default:[],googlecloudbasehook:[],googlecloudstoragehook:[],govern:7,grab:1,grade:3,grai:1,grant:7,graph:[],great:3,greater:1,green:1,grossli:7,group:[1,3,12],gunicorn:[0,6],hack:1,had:[1,4],hadoop:[6,10,12],hadoop_dependency_coordin:1,hand:[12,13],handi:10,handl:[1,2,3],happen:[2,11,14],hard:[1,2],harmless:7,has_dag:1,hash:[1,12],hasn:2,have:[1,2,3,4,6,7,8,9,11,12,14],haven:[1,2,14],hdf:[1,2,6],hdfs_conn_id:1,hdfs_default:1,hdfs_path:
1,hdfscli:1,hdfssensor:1,head:[13,14],header:1,heartbeat:[1,2],heavier:1,hello:[2,8],help:[0,7,8,12],here:[1,2,3,4,6,8,10,11,13,14,15],herebi:7,herein:7,hidden:15,hierarchi:[1,14],high:1,highchart:9,higher:1,highli:3,highlight:14,hipchat:1,hipchat_oper:1,hipchatapioper:1,hipchatapisendroomnotificationoper:1,hit:[1,2],hive2mysqloper:1,hive2sambaoper:[1,6],hive:[1,2,3,6,8,12],hive_cli_conn_id:1,hive_cli_default:1,hive_cli_param:1,hive_conn_id:1,hive_t:1,hiveclihook:1,hiveconf:1,hiveconf_jinja_transl:1,hivemetastorehook:1,hiveoper:[1,2,3,12],hivepartitionsensor:[1,2],hivesensoroper:1,hiveserver2:1,hiveserver2_conn_id:1,hiveserver2_default:1,hiveserver2hook:1,hivetodruidtransf:1,hivetomysqltransf:1,hold:7,home:13,homogen:3,hope:11,horribl:14,host:[1,3,12],hostnam:[0,1,2,3],hour:[1,2,4,11],hourli:[1,4,11],hous:2,how:[1,2,3,4,7,8,11,14,15],howev:[2,7,12],hql:1,html:[1,8,14],html_content:1,http:[1,2,7,8,12,14],http_conn_id:1,http_default:1,httphook:1,httpoper:2,httpsensor:1,human:1,icon:1,
icon_url:1,idea:[],idempot:1,identif:7,identifi:[1,7,14,15],ignor:[0,1],ignore_depend:[0,1],ignore_depends_on_past:[0,1],ignore_first_depends_on_past:[0,1],iii:7,imag:1,immedi:[1,2],imper:3,imperson:12,implement:[1,2],impli:[1,7],improv:7,impyla:1,inabl:7,inact:4,inc:7,incident:7,includ:[0,1,2,3,7,11,14],include_adhoc:[0,1],include_downstream:1,include_exampl:1,include_prior_d:1,include_queu:1,include_upstream:1,inclus:7,incorpor:7,incub:[],incur:7,indemn:7,indemnifi:7,indentifi:0,independ:[1,2],index:1,indic:[1,2,7],indirect:7,individu:[1,7,11,14],infer:[1,2],inferred_op:2,infil:1,infin:5,info:1,inform:[1,2,3,7,11,12,14],infrastructur:1,infring:7,ingest:1,inherit:[1,14],init:3,initctl:3,initdb:[0,3,13],initi:[0,3,12,13],initialis:1,inject:8,input:1,input_format:1,insecur:12,insecurecli:1,insert:[1,2],insert_row:1,insid:[1,2],inspect:[11,13],instal:[],instanc:[],instanti:[],instati:14,instead:[0,1,4],insteadnhttp:1,institut:7,instruct:2,insur:1,integr:[],integrate_plugin:1,intention
:7,inter:2,interact:[1,2,3,8,9],interest:[1,14],intergr:8,intern:0,interpret:[1,2],interv:1,intervalcheckoper:1,introduc:4,intuit:14,invari:[1,2],invok:1,ipython:2,irrevoc:7,is_paus:1,is_prematur:1,is_queueabl:1,is_runn:1,ish:1,isn:[1,2,4],isoformat:1,issu:[1,2,5,7,10],item:2,iter:1,itself:[2,12,14],januari:[7,11],java:1,jdbc:[1,6],jeremiah:10,jinja2:1,jinja:[],jinja_env:1,jira:[],jlowin:10,job:[0,1,2,3,4,11,13,15],job_id:1,jobtrack:1,join:[2,10],json:[0,1,2],jupyt:2,just:[0,1,2,4,8,14],kadmin:12,kapil:3,keep:[],kei:[0,1,2,3,4,12,15],kerber:[6,12],kerbero:[],key_fil:1,key_path:[],keytab:[0,12],keyword:1,kick:[3,11],kill:[1,2],kill_zombi:1,kind:7,know:[3,14],knowledg:8,known:6,known_host:1,kwarg:1,kwd:1,lab:1,label:[1,9],lai:13,lambda:[1,2],land:[1,8],languag:[7,14],larg:1,larger:1,last:1,late:15,later:9,latest:[0,1,6,14],latest_d:1,latest_execution_d:1,launch:12,law:7,lawsuit:7,layout:14,ldap3:12,ldap:[],ldap_auth:12,ldap_ca:12,lead:[1,2],lean:5,least:[1,2,7],leav:1,led:2,left:[2,12
],legaci:4,legal:7,len:1,leo:10,less:1,lesson:10,let:[0,1,2,3,9,14,15],level:[1,2,5,11,12],leverag:[1,2,14],liabil:7,liabl:7,lib:[1,2,3],libffi:3,librari:[1,2,3,5,8,14],libz:2,licens:[],licensor:7,life:2,light:[1,2],lighter:1,lightweight:2,like:[1,2,3,6,8,12,14],limit:[],link:[],linkedin:[],list:[0,1,2,3,4,6,7,8,10,12,14,15],list_dag:[0,4,14],list_directori:1,list_kei:1,list_prefix:1,list_task:[0,4,14],listen:2,lit:1,liter:1,litig:7,live:1,load:[1,2],load_fil:1,load_from_hdf:1,load_str:1,local:[0,1,2,3,12,14],local_executor:1,local_full_path_or_buff:1,local_infil:1,local_port:1,localexecutor:[0,1,2,3,11],localhost:2,locat:[0,1,2,3,12,14],lock:1,lock_for_upd:1,log:[],log_fil:0,loggingmixin:1,logic:[1,2,14],login:[1,2,3,12],logist:8,look:[0,1,2,5,8,10,11,14],loss:7,lot:11,lowin:10,machin:[2,3,12],made:[1,2,7],mai:[1,2,3,4,7,11,14],mail:[7,10],maill:[],main:[1,2,4,8],main_dag:2,maintain:[1,5,10],make:[1,2,3,4,5,7,12,14,15],malfunct:7,malik:3,man:6,manag:[],mani:[0,1,2,4,8,14,15],manual
:[1,2,3,11],map:1,mapr:1,mark:[0,1,2,7,11],mark_success:[0,1,11],markdown:2,markup:1,master:[1,2,14],match:[0,1,2],matter:1,max:[1,10],max_active_run:[1,4],max_partit:1,maxim:10,maxime_beauchemin:0,maximum:1,mayb:2,mean:[1,2,3,4,7,12,14],meaning:1,meant:1,mechan:[1,7,12],media:7,medium:7,meet:[1,7],mem_limit:1,member:12,memberof:12,memori:1,menu:[],menu_link:8,menulink:8,merchant:7,mere:7,merg:14,mesosexecutor:[3,11],messag:[1,2,5],message_format:1,met:[1,3,4,11],metadata:[],metastor:1,metastore_conn_id:1,metastore_default:1,metastore_mysql:1,metastorepartitionsensor:1,method:[1,2],metric:[1,8],metric_spec:1,metrics_threshold:1,microsoft:[1,6],midnight:[4,11],might:[1,2,4,11,14],migrat:12,mime:[],mime_typ:[],min:4,mind:[],minimum:[1,6,11],minut:[2,4,11,14],mismatch:2,miss:[1,2],mistercrunch:10,mix:14,mkdir:2,mlsd:1,mode:[0,1,14],modif:7,modifi:[1,2,3,7,9],modul:[],modular:5,monitor:[2,3,5,8,11,15],monospac:2,month:[4,11],monthli:[4,11],more:[1,2,3,5,7,8,11,12,14,15],morn:11,most:[1,
2,12,14,15],mostli:[1,5,11],mount:[1,3],move:[1,2,4,5],msg:1,mssql:[1,6],mssql_conn_id:1,mssql_default:1,mssqlhook:1,mssqloper:1,mssqltohivetransf:1,much:[1,2,11],multi:[],multipart:1,multipart_byt:1,multipl:[1,2],multipli:1,multiprocess:1,must:[1,2,3,4,7,12,14],mutat:2,my_conn_str:3,my_dag1:2,my_dag2:2,my_dag:2,my_databas:1,my_funct:2,my_param:14,my_tabl:1,my_useful_packag:2,myarg:1,mypackag:12,mys3conn:3,mysql:[1,2,3,6],mysql_conn_id:1,mysql_default:1,mysql_postoper:1,mysql_preoper:1,mysql_tabl:1,mysqldb:1,mysqlhook:1,mysqloper:[1,3],mysqltohivetransf:1,name:[1,2,3,7,8,12],namespac:[1,8],nav:12,navig:12,ndfor:14,necessarili:7,need:[1,2,3,4,5,6,8,11,12,13,14],neg:[1,11],neglig:7,neighbour:1,nerd:[],nest:[1,14],network:[1,12],network_mod:1,never:[0,1,4],new_user_email:12,new_user_nam:12,newobject:1,next:[],nhere:1,nice:2,night:2,nlst:1,no_confirm:0,no_host_key_check:1,node:[0,1,2],non:[1,3,7],none:[1,2,8,11],noram:12,norandkei:12,normal:[1,2,7],notat:1,note:[],notebook:2,noth:[1,7],
notic:[2,7,14],notif:1,notifi:1,notwithstand:7,nov:10,now:[1,2,3,4,12,14],nuanc:2,num:1,num_run:0,num_shard:1,number:[0,1,2,3,5],o2snqeoz7:[],oauth2:12,oauth_callback_rout:12,oauth_key_from_github_enterpris:12,oauth_secret_from_github_enterpris:12,obj:1,object:[0,1,2,3,4,7,8,11,14],objectclass:12,oblig:7,observ:1,obtain:7,occur:[1,2],occurr:1,octet:[],off:[1,3,11],offer:[7,8],offici:[1,10],offset:1,often:[1,2],on_failure_callback:1,on_kil:1,on_retry_callback:1,on_success_callback:1,onboard:4,onc:[1,2,3,4,11,14],one_fail:[1,2],one_success:[1,2],onli:[0,1,2,3,4,6,7,11,12,13],only_fail:0,only_if_upd:1,only_run:0,oozi:5,op1:2,op2:2,op3:2,op4:2,op_arg:1,op_kwarg:1,open:[1,2,4,10,12],oper:[],oppos:[1,2],optim:1,option:[],orchestr:[1,5],order:[1,2,3,11,12],org:[7,8,10,14],organ:[2,8],organiz:12,origin:7,orm:1,other:[1,2,3,5,7,9,11,14],otherwis:[1,7,14],our:14,out:[],outag:8,outgrow:13,outlier:15,output:[1,14],output_encod:1,output_format:1,outsid:11,outstand:7,over:[0,1,2,3,13,15],overlap:
15,overrid:[0,1,14],overridden:[1,4],overview:15,overwhelm:2,overwrit:1,own:[],owner:[1,2,7,12,14],ownership:7,oyp49mbwh60:[],p12:[],packag:[],package1:2,page:[2,4,7,10,15],pai:[1,4],pair:[1,15],panda:1,parallel:[1,2,3,13],param:[0,1,14],paramet:[0,1,2,3,4,9,14],parameter:[2,5,9],parent:2,parent_dag:[1,2],parent_dag_nam:2,pars:[1,4,8,14],part:[1,2,7,9,12],parti:7,particular:7,partit:[1,2],partition_nam:1,partli:4,pass:[1,2,4,8,14],pass_valu:1,passwd:15,password:[],password_auth:12,passwordus:12,past:[1,2,4,11,15],patent:7,path2:1,path:[1,2,3],patrick:10,patrickleotardif:10,pattern:[1,2],paus:[0,1],payload:1,pem:1,peopl:[1,8,14],percent:7,perfect:2,perform:[0,1,2,3,5,7,15],perhap:[2,15],period:[1,2,4,11,14],permiss:[7,12],perpetu:7,persist:[1,11],perspect:2,pertain:7,phase:8,pick:[2,3,9],pickl:[0,1,2,3],picklabl:1,pickle_id:1,pickleabl:1,pid:[0,13],pig:[1,2],piggi:8,pigoper:1,pin_100:1,pip:[2,6,13],pipelin:[],pitfal:10,place:[1,2,7,14],placehold:1,plain:[1,3],plan:4,platform:[2,5,6],
pleas:[2,3,10,12,14],plug:8,plugin:[],pluginexecutor:8,pluginhook:8,pluginoper:8,plugins_manag:8,png:1,point:[1,2,3,4,13,14],poke:1,poke_interv:1,polymorph:1,pool:[],pool_ful:1,popen:1,popul:14,port:[0,1,3,12,13],posit:[0,1],possibl:[0,1,2,3,7,12,13,14,15],post:[1,10],post_execut:1,postgr:[1,2,3,6],postgres_conn_id:1,postgres_default:1,postgres_mast:2,postgreshook:1,postgresoper:1,potenti:2,power:[2,5,7,14],practic:[1,10],pre:4,pre_execut:1,preced:[1,3,14],prefer:[7,11,13],prefix:[1,2,3],preoper:1,prepar:[2,7],prepare_templ:1,prepped_request:1,present:1,preset:11,presto:1,presto_conn_id:1,presto_default:1,prestocheckoper:1,prestohook:1,prestointervalcheckoper:1,prestotomysqloper:2,prestovaluecheckoper:1,pretti:14,prevent:[1,2,12],previou:[0,1,2,4,14],previous:2,primari:1,primit:1,princip:[0,2,12],print:[1,2,7,14],print_dat:14,prior:1,priorit:2,prioriti:[1,2],priority_weight:[1,2,14],privaci:7,probabl:[3,4,12],problem:2,problemat:2,proce:[1,4],process:[0,1,2,3,11,14],process_fil:1,pr
oduct:[1,2,3,5,7,9,11,14],profil:[],program:10,programmat:5,progress:[1,5,14],project:[],promin:7,prompt:0,propag:[1,2],propaget:1,proper:14,properli:[3,4],properti:[1,2,4,12],provid:[1,2,3,7,9,14,15],provide_context:[1,2],proxy_us:[1,12],proxyus:12,psycopg2:3,ptarget:1,publicli:7,publish:1,pull:[1,2],pull_funct:2,puls:[],puppet:[3,10],pure:2,purpl:1,purpos:[7,14],push:[1,2,8],push_funct:2,pushing_task:2,put:[1,2,8],pwd:2,pyhiv:1,pypi:[6,13],python:[1,2,3,5,6,8,12,14],python_1:2,python_cal:[1,2],python_oper:1,pythonhost:8,pythonoper:[1,2],pythonpath:[3,12],qualif:12,qualifi:12,qualiti:[1,2],qubol:6,queri:[],queu:[1,2,4],queue:[],quick:[],quickli:[9,13,14,15],quit:4,r39132:10,rabbitmq:[3,6],rais:[1,14],ran:0,randkei:12,random:1,randomli:2,rang:[0,1,11,14],rather:2,ratio:1,raw:1,reach:[1,2,3,4],reactiv:4,read:[1,3,4,12,14],readabl:[1,7,12],readi:[1,5,13],readm:3,readthedoc:14,ready_for_retri:1,real:3,realli:[1,12,14],realm:12,reason:[4,7,14],reboot:2,rebuild:0,recap:[],receiv:[1,2,7,1
1],recent:[1,2],recip:10,recipi:7,recogn:1,recommend:[1,3,4,7],record:[0,1,2,14],recreat:1,recurs:1,red:1,redhat:3,redi:3,redirect:0,redistribut:7,redund:14,referenc:[2,14],reflect:[2,8,14],refrain:2,refresh:1,refresh_from_db:1,regard:[1,7],regardless:[0,1],regex:[0,1],regex_kei:1,regexp:0,regist:[8,9,12],regular:[1,11],reinit_frequ:12,rel:[1,11,14],relat:[1,2,3,4,6],relationship:2,relativedelta:1,relev:4,reli:[1,12],reliabl:1,remain:[7,12],rememb:[1,14],remot:[1,3,11],remote_base_log_fold:3,remote_full_path:1,remote_host:1,remote_log_conn_id:3,remote_port:1,remov:1,render:[0,1,2,8,14],render_templ:1,render_template_from_field:1,renew:[0,12],repeat:2,repl:12,replac:[1,7,12],report:1,repositori:[3,14],repres:[1,2,4,7,11],represent:[1,15],reproduc:7,reproduct:7,request:[0,1,2],requir:[1,2,3,6,7,12],resembl:2,reset:0,resetdb:0,resid:2,resolv:1,resourc:[],respect:[0,1,2,14],respons:[1,7],response_check:1,rest:1,restart:[2,3,4],restrict:12,restructuredtext:2,result:[1,2,7,9,14],retain:7,
retri:[1,2,14],retriev:[1,2],retrieve_fil:1,retry_delai:[1,14],return_valu:1,reus:8,revis:7,riccomini:10,rich:[0,2,5,15],rid:1,right:[1,2,7,9],rightmost:2,risk:7,role:3,roll:[],room:1,room_id:1,root:[1,2],root_dag:1,round:[1,4],rout:12,routin:[1,2],row:[1,2],royalti:7,rule:[],run:[],run_a:1,run_and_check:1,run_as_own:[1,12],run_cli:1,run_id:[0,1,11],runme_0:13,runnabl:[1,2],rwc:12,s3_conn_id:1,s3_default:1,s3_kei:1,s3connect:1,s3filetransferoper:2,s3hook:[1,3],s3keysensor:[1,6],s3prefixsensor:6,s3tohivetransf:1,safe_mod:1,sai:[1,2,4,11],said:2,samaccountnam:12,samba:[1,6],samba_conn_id:1,samba_default:1,same:[1,2,3,7,9,12],sampl:3,sandbox:3,save:[3,4,9,14],scalabl:5,scan:[1,2],scene:[6,11],schedul:[],schedule_interv:[1,2,4,11,14],scheduler_interv:4,scheduler_run:3,schema:[1,2,3],scratch:14,screen:14,screenshot:[],script:[],script_begin_tag:1,search:[1,2],search_path:3,search_scop:12,second:[1,2,4,14],secret:[12,15],section:[1,2,3,4,7,12,13,14],secur:[],see:[1,2,3,4,7,11,12,13,14,15]
,seen:[3,15],segment:1,select:[1,2,9,11,12],self:[1,8],sell:7,semicolon:1,send:[0,1,2,10],send_room_notif:1,sender:1,sensor:[1,2,4],sensor_queu:2,sent:[1,2,3,7],separ:[2,7,14],sequenti:[1,11,13],sequentialexecutor:[1,2,13],serial:[0,1],serv:[0,3,14],serve_log:[0,3],server:[0,1,3,8,12,13,14],servic:[0,1,2,3,6,7,11,12],service_account:[],session:[1,12],set:[],set_depend:[1,14],set_downstream:[1,2,14],set_the_password:12,set_upstream:[1,2,14],setgid:3,setuid:3,setup:[3,12],sever:2,shall:7,shape:11,share:[1,2,3,7,8,9],shell:12,ship:[0,1],ship_dag:0,shop:7,shortcircuitoper:1,shortcut:[3,15],shortest:1,should:[1,2,3,4,7,8,11,13,14],shouldn:[1,2],show:[0,1,2,4,8,15],shown:1,sid:10,siddharth:10,side:[1,3],signal:1,similar:[2,3,5],simpl:[1,2,8,9,14],simplehttpoper:1,simpler:14,simplest:12,simpli:[1,2,8,11,12,14],simplifi:1,simul:14,simultan:2,sinc:[1,3,13,14],sing:1,singl:[0,1,2,14],site:12,size:[1,2],skew:8,skip:[0,1,2],sla:[],sla_miss_callback:1,slack:[1,6],slack_oper:1,slackapioper:1,slac
kapipostoper:[1,6],slackoper:2,slave:3,sleep:14,slightli:5,slot:[1,2],slowli:5,slug:12,small:[1,2],smaller:1,smallish:1,snap:5,sock:1,soft_fail:1,softwar:[7,10],sole:7,solut:5,some:[1,2,3,4,11,14,15],somehow:3,someth:[2,12,14],sometim:[1,2,14],somewhat:14,somewher:13,soon:[1,2,3,4,11],sort:2,sound:[2,14],sourc:[1,2,7,9,10,15],source_conn_id:1,space:5,span:15,spark:[2,5],spawn:3,special:[1,2,4,7],specif:[0,1,2,3,4,7,11,14,15],specifi:[0,1,2,3,4,5,11,12,14],specific_valu:1,spent:15,spring:10,sql:[1,2,3,6,9],sql_alchemy_conn:3,sql_alchemy_conn_cmd:3,sqlachemi:1,sqlalchemi:[1,2,3],sqlite:[1,13],sqlitehook:1,sqloper:2,sqlsensor:1,squeez:1,sscursor:1,ssh:[1,12],ssh_default:1,ssh_hook:1,sshexecuteoper:1,sshhook:1,sshpass:1,ssl:1,sslcert:1,sslmode:1,stabl:6,stack:[6,8],stage:[1,2],stai:11,stamp:11,stand:2,standalon:2,standard:[1,2],stanza:3,start:[],start_dat:[0,1,2,4,11,14],stat:1,state:[0,1,2,4,7,11,14],state_d:1,statement:[1,2,7],static_babynam:1,static_babynames_partit:1,static_fold:8,s
tatic_path:1,static_url_path:8,statu:[0,1,2,3,4,7,11,13,14,15],stderr:0,stdout:[0,1,14],step:[1,3,14,15],still:[1,4],sting:1,stop:[1,3],stoppag:7,storag:3,store:[1,2,3,8,12,13],store_fil:1,storm:5,str:[1,8,11],straightforward:13,stream:[1,5],string:[0,1,2,3,14],string_data:1,structur:[1,5,14],style:1,sub:[0,1],sub_dag:[1,2],sub_part:1,subcommand:[1,3,11,14],subdag:[],subdagoper:2,subdir:0,subdirectori:2,subject:[1,7],sublicens:7,submiss:[7,11],submit:7,subpackag:[6,12],subpartit:1,subprocess:[1,11],subscrib:[2,10],subsect:[0,11],subsequ:[0,7,11],subset:1,subtl:2,subtract:1,subtre:12,succe:[1,2],succeed:[0,1,2,4,15],success:[0,1,2,4,11,14],successfulli:[1,2,14],suit:5,sum:2,summit:10,sundai:11,super_us:12,supersed:7,superus:12,superuser_filt:12,suppli:[3,12],support:[0,1,3,6,7,12],sure:[1,2,3,4,12,14],surfac:14,surgeri:5,sync:[0,1,3,11],sync_to_db:1,synchron:[1,3],syntax:7,sysconfig:3,system:[1,2,3,7],tab:[1,12],tabl:[1,2,8],table_exist:1,table_nam:[1,2],tablenam:1,tag:4,take:[1,2,3,
10,12,13,14,15],talk:[1,10,14],tardif:10,target:[1,3,11],target_field:1,target_partition_s:1,target_tim:1,task:[],task_id:[0,1,2,11,14],task_inst:[1,2],task_instance_key_str:1,task_or_task_list:1,task_param:0,task_regex:[0,1],task_stat:0,taskinst:[1,2],tbl:1,team:[1,12],tech:[],tell:[0,1],templat:[],template_fold:8,template_searchpath:[1,14],templated_command:14,templates_dict:1,templates_ext:1,temporari:[1,2],tenanc:[],tenant:12,term:7,termin:[2,7],test:[],test_env:2,test_hql:1,test_mod:1,test_plugin:8,testabl:5,testview:8,text:[1,3,7,15],textfil:1,than:[1,2,5,14],thei:[0,1,2,3,5,6,8,9,11,14],them:[0,1,2,8,11,14],themselv:2,theori:[4,7],thereof:7,thi:[0,1,2,3,4,5,7,8,11,12,13,14,15],thin:1,thing:[1,2,4,14],think:[5,14],third:7,this_dag_will_be_discov:2,those:[1,2,3,7,12],though:[2,5],thousand:2,thread:1,three:[2,3],threshold:8,thrift:1,through:[1,2,6,7,11,13,14],throughout:1,ticket:[0,12],tighten:12,time:[1,2,3,4,8,11,12,14,15],timedelta:[1,2,4,11,14],timedeltasensor:[1,4],timefram
:1,timeout:[0,1,2],timesensor:1,timestamp:[1,2,11],tip:[2,10],titl:7,tls_ca_cert:1,tls_client_cert:1,tls_client_kei:1,tls_hostnam:1,tls_ssl_version:1,tmp:[1,2],tmp_dir:1,tmp_file:1,todai:1,togeth:[1,2],token:1,toler:[1,2],tomorrow:1,tomorrow_d:1,tomorrow_ds_nodash:1,too:[1,2,11],tool:[1,2,6,8],toolbox:8,top:[],topic:3,tornado:0,tort:7,tour:[13,14],track:[2,7,14],tracker:1,trade:7,trademark:7,tradit:1,tradition:2,train:7,transact:1,transfer:[1,2,7],transform:7,translat:[1,7],transpar:15,travers:1,treat:[1,2],tree:[],tree_view:1,tri:[1,12],trick:10,trigger:[],trigger_dag:[0,11],trigger_dag_id:1,trigger_rul:[1,2],triggerdagrunoper:[1,10],triggerrul:1,troubl:6,troubleshoot:[5,15],truncat:1,truth:1,ts_dim:1,ts_nodash:1,tsv:1,tty:1,tunnel:[1,12],tupl:1,turn:[2,12],tutori:[],twice:1,two:[1,2,14],type:[0,1,2,4,7,9,12],typic:[1,2],ubuntu:3,uid:12,ultim:[1,14],unassign:2,unavail:3,unblock:11,undefin:12,under:[1,2,3,7,8,10],underli:8,underscor:3,understand:[1,4,8,14,15],undesir:11,unexpect:2,u
nexpectedli:2,union:7,uniqu:[1,14],unit:[3,5,12],unittest:1,unix:1,unless:[4,7],unpack:[1,2],unpaus:0,unset:1,unsound:[1,2],unsuccess:1,until:[1,2,3],up_for_retri:1,updat:[1,2,4,12],upgrad:0,upgradedb:0,upload:1,upon:[2,3,13],upstream:[0,1,2,4,11],upstream_fail:[1,2],upstream_list:1,upstream_task_id:1,uri:[1,2,3,12],url:[1,3,8,9,12],usag:[0,14],use_beelin:[1,12],user:[0,1,2,3,5,6,12,14],user_defined_macro:1,user_filt:12,user_name_attr:12,usernam:[1,3,12],usr:3,usual:[2,3],utf8:1,utf:1,util:[1,5,13],uuid:1,val:[0,1],valid:[],valu:[0,1,2,3,4,12,14,15],valuecheckoper:1,variabl:[],vast:1,verbal:7,verbos:[1,14],veri:[0,1,2,4,10,13,14],verifi:[2,4],version:[0,1,2,5,6,7],vertia:1,vertica:[1,6],vertica_conn_id:1,vertica_default:1,verticahook:1,verticaoper:1,verticaql:1,verticatohivetransf:1,via:[1,2,12],video:[1,10],view:[],virtualenv:2,visibl:[1,2],visual:[5,8,9,14,15],vocabulari:2,volum:1,wai:[1,2,3,4,6,8,11,12,14,15],wait:[0,1,2],wait_for_downstream:[1,4],wait_for_empty_queu:2,walk:14,wa
nt:[1,2,3,4,8,11,12,14],warranti:7,wasn:4,wasnt:1,watch:[1,3,4],weapon:9,web:[],webhdfs_conn_id:1,webhdfs_default:1,webhdfshook:1,webhdfssensor:1,webserv:[0,3,4,12,13,14],week:11,weekli:11,weight:1,well:[1,2,3,4,11],were:1,what:[],whatev:[2,3,6],when:[1,2,3,4,5,8,9,11,12,14],whenev:1,where:[1,2,3,4,7,11,12,14,15],wherev:[1,7],whether:[1,7,11],which:[0,1,2,3,4,7,11,12,13,14],whitelist:12,who:8,whole:[1,2,7],whom:7,whose:11,why:[],wiedmer:10,wiki:10,wildcard:1,wildcard_kei:1,wildcard_match:1,win:1,window:11,winter:10,wire:2,within:[0,1,2,3,4,7],without:[0,1,2,7,11,12],won:[1,4,6],word:[2,3,11,15],work:[0,1,2,3,4,5,7,8,9,11,12,13],worker:[0,1,2,3,4,5,6,8,12,14],worker_timeout:0,workerclass:0,workflow:[],workload:8,worldwid:7,worri:14,worth:1,would:[1,2,3,4,5,11,12,14],wouldn:[1,4],wrap:14,wrapper:1,write:[3,5,7,8,9,14],written:[1,7,14],wrong:14,www:[1,7],x01:1,xcom:[],xcom_al:1,xcom_pul:[1,2],xcom_push:[1,2],xcom_return_kei:1,xml:12,xst:12,yaml:2,year:[1,11],yearli:11,yellow:1,yesterda
i:1,yesterday_d:1,yesterday_ds_nodash:1,yet:[1,2,12,15],you:[1,2,3,4,5,6,7,8,9,11,12,13,14,15],your:[],yourself:1,youtu:[],youtub:1,yum:6,yyyi:[0,1,2],yyyymmdd:1,zero:1,zip:[1,2],zip_dag:2,zip_dag_cont:2,zoom:2},titles:["Command Line Interface","API Reference","Concepts","Configuration","FAQ","Apache Airflow (incubating) Documentation","Installation","License","Plugins","Data Profiling","Project","Scheduling & Triggers","Security","Quick Start","Tutorial","UI / Screenshots"],titleterms:{"default":[1,2,14],"function":2,"import":14,addit:2,adhoc:9,airflow:[5,6,8],apach:5,api:1,argument:[2,14],assign:2,authent:12,backend:3,backfil:14,baseoper:1,basesensoroper:1,beyond:5,bitshift:2,branch:2,build:8,celeri:3,chart:[9,15],cluster:2,code:15,command:[0,14],committ:10,commun:[1,3],composit:2,concept:2,configur:3,connect:[2,3],content:5,context:[2,15],contribut:[1,3],core:2,dag:[2,11,14,15],data:9,definit:14,depend:14,document:[2,5],durat:15,enabl:12,enterpris:12,exampl:[8,14],executor:1,
extern:11,extra:6,faq:4,file:14,form:9,gantt:15,get:6,ghe:12,github:12,graph:15,histori:10,hook:[1,2],horizon:5,idea:2,incub:5,instal:6,instanc:[2,15],instanti:14,integr:3,interfac:[0,8],jinja:[2,14],keep:11,kerbero:12,ldap:12,licens:7,limit:12,line:[0,14],link:10,log:3,macro:1,manag:2,menu:15,meso:3,metadata:14,mind:11,model:1,modul:14,multi:12,next:[13,14],note:2,oper:[1,2],option:3,out:3,own:12,packag:[2,6],password:12,pipelin:14,plugin:8,polici:2,pool:2,principl:5,profil:9,project:10,queri:9,queue:2,quick:13,recap:14,refer:1,resourc:10,roadmap:10,roll:12,rule:2,run:[11,14],scale:3,schedul:11,scope:2,screenshot:[9,15],script:14,secur:12,set:[3,12,14],sla:2,start:13,subdag:2,systemd:3,task:[2,14,15],templat:[2,14],tenanc:12,test:14,top:8,tree:15,trigger:[2,11],tutori:14,undead:2,upstart:3,valid:14,variabl:[1,2,15],view:15,web:12,what:[8,13,14],why:8,workflow:2,xcom:2,your:12,zombi:2}})
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/security.html
----------------------------------------------------------------------
diff --git a/security.html b/security.html
new file mode 100644
index 0000000..7ca3bf7
--- /dev/null
+++ b/security.html
@@ -0,0 +1,436 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Security — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="FAQ" href="faq.html"/>
+ <link rel="prev" title="Plugins" href="plugins.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">Security</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#web-authentication">Web Authentication</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#password">Password</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#ldap">LDAP</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#roll-your-own">Roll your own</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="#multi-tenancy">Multi-tenancy</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#kerberos">Kerberos</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#limitations">Limitations</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#enabling-kerberos">Enabling kerberos</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#using-kerberos-authentication">Using kerberos authentication</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#github-enterprise-ghe-authentication">GitHub Enterprise (GHE) Authentication</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#setting-up-ghe-authentication">Setting up GHE Authentication</a></li>
+</ul>
+</li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>Security</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/security.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="security">
+<h1>Security<a class="headerlink" href="#security" title="Permalink to this headline">�</a></h1>
+<div class="section" id="web-authentication">
+<h2>Web Authentication<a class="headerlink" href="#web-authentication" title="Permalink to this headline">�</a></h2>
+<p>By default, all gates are opened. An easy way to restrict access
+to the web application is to do it at the network level, or by using
+SSH tunnels.</p>
+<p>It is however possible to switch on authentication by either using one of the supplied
+backends or create your own.</p>
+<div class="section" id="password">
+<h3>Password<a class="headerlink" href="#password" title="Permalink to this headline">�</a></h3>
+<p>One of the simplest mechanisms for authentication is requiring users to specify a password before logging in.
+Password authentication requires the used of the <code class="docutils literal"><span class="pre">password</span></code> subpackage in your requirements file. Password hashing
+uses bcrypt before storing passwords.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
+<span class="nv">authenticate</span> <span class="o">=</span> True
+<span class="nv">auth_backend</span> <span class="o">=</span> airflow.contrib.auth.backends.password_auth
+</pre></div>
+</div>
+<p>When password auth is enabled, an initial user credential will need to be created before anyone can login. An initial
+user was not created in the migrations for this authenication backend to prevent default Airflow installations from
+attack. Creating a new user has to be done via a Python REPL on the same machine Airflow is installed.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># navigate to the airflow installation directory</span>
+$ <span class="nb">cd</span> ~/airflow
+$ python
+Python 2.7.9 <span class="o">(</span>default, Feb <span class="m">10</span> 2015, 03:28:08<span class="o">)</span>
+Type <span class="s2">"help"</span>, <span class="s2">"copyright"</span>, <span class="s2">"credits"</span> or <span class="s2">"license"</span> <span class="k">for</span> more information.
+>>> import airflow
+>>> from airflow import models, settings
+>>> from airflow.contrib.auth.backends.password_auth import PasswordUser
+>>> <span class="nv">user</span> <span class="o">=</span> PasswordUser<span class="o">(</span>models.User<span class="o">())</span>
+>>> user.username <span class="o">=</span> <span class="s1">'new_user_name'</span>
+>>> user.email <span class="o">=</span> <span class="s1">'new_user_email@example.com'</span>
+>>> user.password <span class="o">=</span> <span class="s1">'set_the_password'</span>
+>>> <span class="nv">session</span> <span class="o">=</span> settings.Session<span class="o">()</span>
+>>> session.add<span class="o">(</span>user<span class="o">)</span>
+>>> session.commit<span class="o">()</span>
+>>> session.close<span class="o">()</span>
+>>> exit<span class="o">()</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="ldap">
+<h3>LDAP<a class="headerlink" href="#ldap" title="Permalink to this headline">�</a></h3>
+<p>To turn on LDAP authentication configure your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> as follows. Please note that the example uses
+an encrypted connection to the ldap server as you probably do not want passwords be readable on the network level.
+It is however possible to configure without encryption if you really want to.</p>
+<p>Additionally, if you are using Active Directory, and are not explicitly specifying an OU that your users are in,
+you will need to change <code class="docutils literal"><span class="pre">search_scope</span></code> to “SUBTREE”.</p>
+<p>Valid search_scope options can be found in the <a class="reference external" href="http://ldap3.readthedocs.org/searches.html?highlight=search_scope">ldap3 Documentation</a></p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
+<span class="nv">authenticate</span> <span class="o">=</span> True
+<span class="nv">auth_backend</span> <span class="o">=</span> airflow.contrib.auth.backends.ldap_auth
+
+<span class="o">[</span>ldap<span class="o">]</span>
+<span class="nv">uri</span> <span class="o">=</span> ldaps://<your.ldap.server>:<port>
+<span class="nv">user_filter</span> <span class="o">=</span> <span class="nv">objectClass</span><span class="o">=</span>*
+<span class="nv">user_name_attr</span> <span class="o">=</span> uid <span class="c1"># in case of Active Directory you would use sAMAccountName</span>
+<span class="nv">superuser_filter</span> <span class="o">=</span> <span class="nv">memberOf</span><span class="o">=</span><span class="nv">CN</span><span class="o">=</span>airflow-super-users,OU<span class="o">=</span>Groups,OU<span class="o">=</span>RWC,OU<span class="o">=</span>US,OU<span class="o">=</span>NORAM,DC<span class="o">=</span>example,DC<span class="o">=</span>com
+<span class="nv">data_profiler_filter</span> <span class="o">=</span> <span class="nv">memberOf</span><span class="o">=</span><span class="nv">CN</span><span class="o">=</span>airflow-data-profilers,OU<span class="o">=</span>Groups,OU<span class="o">=</span>RWC,OU<span class="o">=</span>US,OU<span class="o">=</span>NORAM,DC<span class="o">=</span>example,DC<span class="o">=</span>com
+<span class="nv">bind_user</span> <span class="o">=</span> <span class="nv">cn</span><span class="o">=</span>Manager,dc<span class="o">=</span>example,dc<span class="o">=</span>com
+<span class="nv">bind_password</span> <span class="o">=</span> insecure
+<span class="nv">basedn</span> <span class="o">=</span> <span class="nv">dc</span><span class="o">=</span>example,dc<span class="o">=</span>com
+<span class="nv">cacert</span> <span class="o">=</span> /etc/ca/ldap_ca.crt
+<span class="nv">search_scope</span> <span class="o">=</span> LEVEL <span class="c1"># Set this to SUBTREE if using Active Directory, and not specifying an Organizational Unit</span>
+</pre></div>
+</div>
+<p>The superuser_filter and data_profiler_filter are optional. If defined, these configurations allow you to specify LDAP groups that users must belong to in order to have superuser (admin) and data-profiler permissions. If undefined, all users will be superusers and data profilers.</p>
+</div>
+<div class="section" id="roll-your-own">
+<h3>Roll your own<a class="headerlink" href="#roll-your-own" title="Permalink to this headline">�</a></h3>
+<p>Airflow uses <code class="docutils literal"><span class="pre">flask_login</span></code> and
+exposes a set of hooks in the <code class="docutils literal"><span class="pre">airflow.default_login</span></code> module. You can
+alter the content and make it part of the <code class="docutils literal"><span class="pre">PYTHONPATH</span></code> and configure it as a backend in <code class="docutils literal"><span class="pre">airflow.cfg`</span></code>.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
+<span class="nv">authenticate</span> <span class="o">=</span> True
+<span class="nv">auth_backend</span> <span class="o">=</span> mypackage.auth
+</pre></div>
+</div>
+</div>
+</div>
+<div class="section" id="multi-tenancy">
+<h2>Multi-tenancy<a class="headerlink" href="#multi-tenancy" title="Permalink to this headline">�</a></h2>
+<p>You can filter the list of dags in webserver by owner name, when authentication
+is turned on, by setting webserver.filter_by_owner as true in your <code class="docutils literal"><span class="pre">airflow.cfg</span></code>
+With this, when a user authenticates and logs into webserver, it will see only the dags
+which it is owner of. A super_user, will be able to see all the dags although.
+This makes the web UI a multi-tenant UI, where a user will only be able to see dags
+created by itself.</p>
+</div>
+<div class="section" id="kerberos">
+<h2>Kerberos<a class="headerlink" href="#kerberos" title="Permalink to this headline">�</a></h2>
+<p>Airflow has initial support for Kerberos. This means that airflow can renew kerberos
+tickets for itself and store it in the ticket cache. The hooks and dags can make use of ticket
+to authenticate against kerberized services.</p>
+<div class="section" id="limitations">
+<h3>Limitations<a class="headerlink" href="#limitations" title="Permalink to this headline">�</a></h3>
+<p>Please note that at this time not all hooks have been adjusted to make use of this functionality yet.
+Also it does not integrate kerberos into the web interface and you will have to rely on network
+level security for now to make sure your service remains secure.</p>
+<p>Celery integration has not been tried and tested yet. However if you generate a key tab for every host
+and launch a ticket renewer next to every worker it will most likely work.</p>
+</div>
+<div class="section" id="enabling-kerberos">
+<h3>Enabling kerberos<a class="headerlink" href="#enabling-kerberos" title="Permalink to this headline">�</a></h3>
+<p>#### Airflow</p>
+<p>To enable kerberos you will need to generate a (service) key tab.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># in the kadmin.local or kadmin shell, create the airflow principal</span>
+kadmin: addprinc -randkey airflow/fully.qualified.domain.name@YOUR-REALM.COM
+
+<span class="c1"># Create the airflow keytab file that will contain the airflow principal</span>
+kadmin: xst -norandkey -k airflow.keytab airflow/fully.qualified.domain.name
+</pre></div>
+</div>
+<p>Now store this file in a location where the airflow user can read it (chmod 600). And then add the following to
+your <code class="docutils literal"><span class="pre">airflow.cfg</span></code></p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>core<span class="o">]</span>
+<span class="nv">security</span> <span class="o">=</span> kerberos
+
+<span class="o">[</span>kerberos<span class="o">]</span>
+<span class="nv">keytab</span> <span class="o">=</span> /etc/airflow/airflow.keytab
+<span class="nv">reinit_frequency</span> <span class="o">=</span> 3600
+<span class="nv">principal</span> <span class="o">=</span> airflow
+</pre></div>
+</div>
+<p>Launch the ticket renewer by</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># run ticket renewer</span>
+airflow kerberos
+</pre></div>
+</div>
+<p>#### Hadoop</p>
+<p>If want to use impersonation this needs to be enabled in <code class="docutils literal"><span class="pre">core-site.xml</span></code> of your hadoop config.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><property>
+ <name>hadoop.proxyuser.airflow.groups</name>
+ <value>*</value>
+</property>
+
+<property>
+ <name>hadoop.proxyuser.airflow.users</name>
+ <value>*</value>
+</property>
+
+<property>
+ <name>hadoop.proxyuser.airflow.hosts</name>
+ <value>*</value>
+</property>
+</pre></div>
+</div>
+<p>Of course if you need to tighten your security replace the asterisk with something more appropriate.</p>
+</div>
+<div class="section" id="using-kerberos-authentication">
+<h3>Using kerberos authentication<a class="headerlink" href="#using-kerberos-authentication" title="Permalink to this headline">�</a></h3>
+<p>The hive hook has been updated to take advantage of kerberos authentication. To allow your DAGs to use it simply
+update the connection details with, for example:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">{</span> <span class="s2">"use_beeline"</span>: true, <span class="s2">"principal"</span>: <span class="s2">"hive/_HOST@EXAMPLE.COM"</span><span class="o">}</span>
+</pre></div>
+</div>
+<p>Adjust the principal to your settings. The _HOST part will be replaced by the fully qualified domain name of
+the server.</p>
+<p>You can specify if you would like to use the dag owner as the user for the connection or the user specified in the login
+section of the connection. For the login user specify the following as extra:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">{</span> <span class="s2">"use_beeline"</span>: true, <span class="s2">"principal"</span>: <span class="s2">"hive/_HOST@EXAMPLE.COM"</span>, <span class="s2">"proxy_user"</span>: <span class="s2">"login"</span><span class="o">}</span>
+</pre></div>
+</div>
+<p>For the DAG owner use:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">{</span> <span class="s2">"use_beeline"</span>: true, <span class="s2">"principal"</span>: <span class="s2">"hive/_HOST@EXAMPLE.COM"</span>, <span class="s2">"proxy_user"</span>: <span class="s2">"owner"</span><span class="o">}</span>
+</pre></div>
+</div>
+<p>and in your DAG, when initializing the HiveOperator, specify</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nv">run_as_owner</span><span class="o">=</span>True
+</pre></div>
+</div>
+</div>
+<div class="section" id="github-enterprise-ghe-authentication">
+<h3>GitHub Enterprise (GHE) Authentication<a class="headerlink" href="#github-enterprise-ghe-authentication" title="Permalink to this headline">�</a></h3>
+<p>The GitHub Enterprise authentication backend can be used to authenticate users
+against an installation of GitHub Enterprise using OAuth2. You can optionally
+specify a team whitelist (composed of slug cased team names) to restrict login
+to only members of those teams.</p>
+<p><em>NOTE</em> If you do not specify a team whitelist, anyone with a valid account on
+your GHE installation will be able to login to Airflow.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
+<span class="nv">authenticate</span> <span class="o">=</span> True
+<span class="nv">auth_backend</span> <span class="o">=</span> airflow.contrib.auth.backends.github_enterprise_auth
+
+<span class="o">[</span>github_enterprise<span class="o">]</span>
+<span class="nv">host</span> <span class="o">=</span> github.example.com
+<span class="nv">client_id</span> <span class="o">=</span> oauth_key_from_github_enterprise
+<span class="nv">client_secret</span> <span class="o">=</span> oauth_secret_from_github_enterprise
+<span class="nv">oauth_callback_route</span> <span class="o">=</span> /example/ghe_oauth/callback
+<span class="nv">allowed_teams</span> <span class="o">=</span> example_team_1, example_team_2
+</pre></div>
+</div>
+</div>
+<div class="section" id="setting-up-ghe-authentication">
+<h3>Setting up GHE Authentication<a class="headerlink" href="#setting-up-ghe-authentication" title="Permalink to this headline">�</a></h3>
+<p>An application must be setup in GHE before you can use the GHE authentication
+backend. In order to setup an application:</p>
+<ol class="arabic simple">
+<li>Navigate to your GHE profile</li>
+<li>Select ‘Applications’ from the left hand nav</li>
+<li>Select the ‘Developer Applications’ tab</li>
+<li>Click ‘Register new application’</li>
+<li>Fill in the required information (the ‘Authorization callback URL’ must be fully qualifed e.g. <a class="reference external" href="http://airflow.example.com/example/ghe_oauth/callback">http://airflow.example.com/example/ghe_oauth/callback</a>)</li>
+<li>Click ‘Register application’</li>
+<li>Copy ‘Client ID’, ‘Client Secret’, and your callback route to your airflow.cfg according to the above example</li>
+</ol>
+</div>
+</div>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="faq.html" class="btn btn-neutral float-right" title="FAQ" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="plugins.html" class="btn btn-neutral" title="Plugins" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[10/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/jquery.js
----------------------------------------------------------------------
diff --git a/_static/jquery.js b/_static/jquery.js
new file mode 100644
index 0000000..ab28a24
--- /dev/null
+++ b/_static/jquery.js
@@ -0,0 +1,4 @@
+/*! jQuery v1.11.1 | (c) 2005, 2014 jQuery Foundation, Inc. | jquery.org/license */
+!function(a,b){"object"==typeof module&&"object"==typeof module.exports?module.exports=a.document?b(a,!0):function(a){if(!a.document)throw new Error("jQuery requires a window with a document");return b(a)}:b(a)}("undefined"!=typeof window?window:this,function(a,b){var c=[],d=c.slice,e=c.concat,f=c.push,g=c.indexOf,h={},i=h.toString,j=h.hasOwnProperty,k={},l="1.11.1",m=function(a,b){return new m.fn.init(a,b)},n=/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g,o=/^-ms-/,p=/-([\da-z])/gi,q=function(a,b){return b.toUpperCase()};m.fn=m.prototype={jquery:l,constructor:m,selector:"",length:0,toArray:function(){return d.call(this)},get:function(a){return null!=a?0>a?this[a+this.length]:this[a]:d.call(this)},pushStack:function(a){var b=m.merge(this.constructor(),a);return b.prevObject=this,b.context=this.context,b},each:function(a,b){return m.each(this,a,b)},map:function(a){return this.pushStack(m.map(this,function(b,c){return a.call(b,c,b)}))},slice:function(){return this.pushStack(d.apply(this,argumen
ts))},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},eq:function(a){var b=this.length,c=+a+(0>a?b:0);return this.pushStack(c>=0&&b>c?[this[c]]:[])},end:function(){return this.prevObject||this.constructor(null)},push:f,sort:c.sort,splice:c.splice},m.extend=m.fn.extend=function(){var a,b,c,d,e,f,g=arguments[0]||{},h=1,i=arguments.length,j=!1;for("boolean"==typeof g&&(j=g,g=arguments[h]||{},h++),"object"==typeof g||m.isFunction(g)||(g={}),h===i&&(g=this,h--);i>h;h++)if(null!=(e=arguments[h]))for(d in e)a=g[d],c=e[d],g!==c&&(j&&c&&(m.isPlainObject(c)||(b=m.isArray(c)))?(b?(b=!1,f=a&&m.isArray(a)?a:[]):f=a&&m.isPlainObject(a)?a:{},g[d]=m.extend(j,f,c)):void 0!==c&&(g[d]=c));return g},m.extend({expando:"jQuery"+(l+Math.random()).replace(/\D/g,""),isReady:!0,error:function(a){throw new Error(a)},noop:function(){},isFunction:function(a){return"function"===m.type(a)},isArray:Array.isArray||function(a){return"array"===m.type(a)},isWindow:function(a){return null!=a&&a=
=a.window},isNumeric:function(a){return!m.isArray(a)&&a-parseFloat(a)>=0},isEmptyObject:function(a){var b;for(b in a)return!1;return!0},isPlainObject:function(a){var b;if(!a||"object"!==m.type(a)||a.nodeType||m.isWindow(a))return!1;try{if(a.constructor&&!j.call(a,"constructor")&&!j.call(a.constructor.prototype,"isPrototypeOf"))return!1}catch(c){return!1}if(k.ownLast)for(b in a)return j.call(a,b);for(b in a);return void 0===b||j.call(a,b)},type:function(a){return null==a?a+"":"object"==typeof a||"function"==typeof a?h[i.call(a)]||"object":typeof a},globalEval:function(b){b&&m.trim(b)&&(a.execScript||function(b){a.eval.call(a,b)})(b)},camelCase:function(a){return a.replace(o,"ms-").replace(p,q)},nodeName:function(a,b){return a.nodeName&&a.nodeName.toLowerCase()===b.toLowerCase()},each:function(a,b,c){var d,e=0,f=a.length,g=r(a);if(c){if(g){for(;f>e;e++)if(d=b.apply(a[e],c),d===!1)break}else for(e in a)if(d=b.apply(a[e],c),d===!1)break}else if(g){for(;f>e;e++)if(d=b.call(a[e],e,a[e]),d
===!1)break}else for(e in a)if(d=b.call(a[e],e,a[e]),d===!1)break;return a},trim:function(a){return null==a?"":(a+"").replace(n,"")},makeArray:function(a,b){var c=b||[];return null!=a&&(r(Object(a))?m.merge(c,"string"==typeof a?[a]:a):f.call(c,a)),c},inArray:function(a,b,c){var d;if(b){if(g)return g.call(b,a,c);for(d=b.length,c=c?0>c?Math.max(0,d+c):c:0;d>c;c++)if(c in b&&b[c]===a)return c}return-1},merge:function(a,b){var c=+b.length,d=0,e=a.length;while(c>d)a[e++]=b[d++];if(c!==c)while(void 0!==b[d])a[e++]=b[d++];return a.length=e,a},grep:function(a,b,c){for(var d,e=[],f=0,g=a.length,h=!c;g>f;f++)d=!b(a[f],f),d!==h&&e.push(a[f]);return e},map:function(a,b,c){var d,f=0,g=a.length,h=r(a),i=[];if(h)for(;g>f;f++)d=b(a[f],f,c),null!=d&&i.push(d);else for(f in a)d=b(a[f],f,c),null!=d&&i.push(d);return e.apply([],i)},guid:1,proxy:function(a,b){var c,e,f;return"string"==typeof b&&(f=a[b],b=a,a=f),m.isFunction(a)?(c=d.call(arguments,2),e=function(){return a.apply(b||this,c.concat(d.call(ar
guments)))},e.guid=a.guid=a.guid||m.guid++,e):void 0},now:function(){return+new Date},support:k}),m.each("Boolean Number String Function Array Date RegExp Object Error".split(" "),function(a,b){h["[object "+b+"]"]=b.toLowerCase()});function r(a){var b=a.length,c=m.type(a);return"function"===c||m.isWindow(a)?!1:1===a.nodeType&&b?!0:"array"===c||0===b||"number"==typeof b&&b>0&&b-1 in a}var s=function(a){var b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u="sizzle"+-new Date,v=a.document,w=0,x=0,y=gb(),z=gb(),A=gb(),B=function(a,b){return a===b&&(l=!0),0},C="undefined",D=1<<31,E={}.hasOwnProperty,F=[],G=F.pop,H=F.push,I=F.push,J=F.slice,K=F.indexOf||function(a){for(var b=0,c=this.length;c>b;b++)if(this[b]===a)return b;return-1},L="checked|selected|async|autofocus|autoplay|controls|defer|disabled|hidden|ismap|loop|multiple|open|readonly|required|scoped",M="[\\x20\\t\\r\\n\\f]",N="(?:\\\\.|[\\w-]|[^\\x00-\\xa0])+",O=N.replace("w","w#"),P="\\["+M+"*("+N+")(?:"+M+"*([*^$|!~]?=)"+M+"*(?:'((?:\\\\.|[
^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\"|("+O+"))|)"+M+"*\\]",Q=":("+N+")(?:\\((('((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\")|((?:\\\\.|[^\\\\()[\\]]|"+P+")*)|.*)\\)|)",R=new RegExp("^"+M+"+|((?:^|[^\\\\])(?:\\\\.)*)"+M+"+$","g"),S=new RegExp("^"+M+"*,"+M+"*"),T=new RegExp("^"+M+"*([>+~]|"+M+")"+M+"*"),U=new RegExp("="+M+"*([^\\]'\"]*?)"+M+"*\\]","g"),V=new RegExp(Q),W=new RegExp("^"+O+"$"),X={ID:new RegExp("^#("+N+")"),CLASS:new RegExp("^\\.("+N+")"),TAG:new RegExp("^("+N.replace("w","w*")+")"),ATTR:new RegExp("^"+P),PSEUDO:new RegExp("^"+Q),CHILD:new RegExp("^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\("+M+"*(even|odd|(([+-]|)(\\d*)n|)"+M+"*(?:([+-]|)"+M+"*(\\d+)|))"+M+"*\\)|)","i"),bool:new RegExp("^(?:"+L+")$","i"),needsContext:new RegExp("^"+M+"*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\("+M+"*((?:-\\d)?\\d*)"+M+"*\\)|)(?=[^-]|$)","i")},Y=/^(?:input|select|textarea|button)$/i,Z=/^h\d$/i,$=/^[^{]+\{\s*\[native \w/,_=/^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/,ab=/[+~]/,
bb=/'|\\/g,cb=new RegExp("\\\\([\\da-f]{1,6}"+M+"?|("+M+")|.)","ig"),db=function(a,b,c){var d="0x"+b-65536;return d!==d||c?b:0>d?String.fromCharCode(d+65536):String.fromCharCode(d>>10|55296,1023&d|56320)};try{I.apply(F=J.call(v.childNodes),v.childNodes),F[v.childNodes.length].nodeType}catch(eb){I={apply:F.length?function(a,b){H.apply(a,J.call(b))}:function(a,b){var c=a.length,d=0;while(a[c++]=b[d++]);a.length=c-1}}}function fb(a,b,d,e){var f,h,j,k,l,o,r,s,w,x;if((b?b.ownerDocument||b:v)!==n&&m(b),b=b||n,d=d||[],!a||"string"!=typeof a)return d;if(1!==(k=b.nodeType)&&9!==k)return[];if(p&&!e){if(f=_.exec(a))if(j=f[1]){if(9===k){if(h=b.getElementById(j),!h||!h.parentNode)return d;if(h.id===j)return d.push(h),d}else if(b.ownerDocument&&(h=b.ownerDocument.getElementById(j))&&t(b,h)&&h.id===j)return d.push(h),d}else{if(f[2])return I.apply(d,b.getElementsByTagName(a)),d;if((j=f[3])&&c.getElementsByClassName&&b.getElementsByClassName)return I.apply(d,b.getElementsByClassName(j)),d}if(c.qsa&&
(!q||!q.test(a))){if(s=r=u,w=b,x=9===k&&a,1===k&&"object"!==b.nodeName.toLowerCase()){o=g(a),(r=b.getAttribute("id"))?s=r.replace(bb,"\\$&"):b.setAttribute("id",s),s="[id='"+s+"'] ",l=o.length;while(l--)o[l]=s+qb(o[l]);w=ab.test(a)&&ob(b.parentNode)||b,x=o.join(",")}if(x)try{return I.apply(d,w.querySelectorAll(x)),d}catch(y){}finally{r||b.removeAttribute("id")}}}return i(a.replace(R,"$1"),b,d,e)}function gb(){var a=[];function b(c,e){return a.push(c+" ")>d.cacheLength&&delete b[a.shift()],b[c+" "]=e}return b}function hb(a){return a[u]=!0,a}function ib(a){var b=n.createElement("div");try{return!!a(b)}catch(c){return!1}finally{b.parentNode&&b.parentNode.removeChild(b),b=null}}function jb(a,b){var c=a.split("|"),e=a.length;while(e--)d.attrHandle[c[e]]=b}function kb(a,b){var c=b&&a,d=c&&1===a.nodeType&&1===b.nodeType&&(~b.sourceIndex||D)-(~a.sourceIndex||D);if(d)return d;if(c)while(c=c.nextSibling)if(c===b)return-1;return a?1:-1}function lb(a){return function(b){var c=b.nodeName.toLower
Case();return"input"===c&&b.type===a}}function mb(a){return function(b){var c=b.nodeName.toLowerCase();return("input"===c||"button"===c)&&b.type===a}}function nb(a){return hb(function(b){return b=+b,hb(function(c,d){var e,f=a([],c.length,b),g=f.length;while(g--)c[e=f[g]]&&(c[e]=!(d[e]=c[e]))})})}function ob(a){return a&&typeof a.getElementsByTagName!==C&&a}c=fb.support={},f=fb.isXML=function(a){var b=a&&(a.ownerDocument||a).documentElement;return b?"HTML"!==b.nodeName:!1},m=fb.setDocument=function(a){var b,e=a?a.ownerDocument||a:v,g=e.defaultView;return e!==n&&9===e.nodeType&&e.documentElement?(n=e,o=e.documentElement,p=!f(e),g&&g!==g.top&&(g.addEventListener?g.addEventListener("unload",function(){m()},!1):g.attachEvent&&g.attachEvent("onunload",function(){m()})),c.attributes=ib(function(a){return a.className="i",!a.getAttribute("className")}),c.getElementsByTagName=ib(function(a){return a.appendChild(e.createComment("")),!a.getElementsByTagName("*").length}),c.getElementsByClassNam
e=$.test(e.getElementsByClassName)&&ib(function(a){return a.innerHTML="<div class='a'></div><div class='a i'></div>",a.firstChild.className="i",2===a.getElementsByClassName("i").length}),c.getById=ib(function(a){return o.appendChild(a).id=u,!e.getElementsByName||!e.getElementsByName(u).length}),c.getById?(d.find.ID=function(a,b){if(typeof b.getElementById!==C&&p){var c=b.getElementById(a);return c&&c.parentNode?[c]:[]}},d.filter.ID=function(a){var b=a.replace(cb,db);return function(a){return a.getAttribute("id")===b}}):(delete d.find.ID,d.filter.ID=function(a){var b=a.replace(cb,db);return function(a){var c=typeof a.getAttributeNode!==C&&a.getAttributeNode("id");return c&&c.value===b}}),d.find.TAG=c.getElementsByTagName?function(a,b){return typeof b.getElementsByTagName!==C?b.getElementsByTagName(a):void 0}:function(a,b){var c,d=[],e=0,f=b.getElementsByTagName(a);if("*"===a){while(c=f[e++])1===c.nodeType&&d.push(c);return d}return f},d.find.CLASS=c.getElementsByClassName&&function(a
,b){return typeof b.getElementsByClassName!==C&&p?b.getElementsByClassName(a):void 0},r=[],q=[],(c.qsa=$.test(e.querySelectorAll))&&(ib(function(a){a.innerHTML="<select msallowclip=''><option selected=''></option></select>",a.querySelectorAll("[msallowclip^='']").length&&q.push("[*^$]="+M+"*(?:''|\"\")"),a.querySelectorAll("[selected]").length||q.push("\\["+M+"*(?:value|"+L+")"),a.querySelectorAll(":checked").length||q.push(":checked")}),ib(function(a){var b=e.createElement("input");b.setAttribute("type","hidden"),a.appendChild(b).setAttribute("name","D"),a.querySelectorAll("[name=d]").length&&q.push("name"+M+"*[*^$|!~]?="),a.querySelectorAll(":enabled").length||q.push(":enabled",":disabled"),a.querySelectorAll("*,:x"),q.push(",.*:")})),(c.matchesSelector=$.test(s=o.matches||o.webkitMatchesSelector||o.mozMatchesSelector||o.oMatchesSelector||o.msMatchesSelector))&&ib(function(a){c.disconnectedMatch=s.call(a,"div"),s.call(a,"[s!='']:x"),r.push("!=",Q)}),q=q.length&&new RegExp(q.join("
|")),r=r.length&&new RegExp(r.join("|")),b=$.test(o.compareDocumentPosition),t=b||$.test(o.contains)?function(a,b){var c=9===a.nodeType?a.documentElement:a,d=b&&b.parentNode;return a===d||!(!d||1!==d.nodeType||!(c.contains?c.contains(d):a.compareDocumentPosition&&16&a.compareDocumentPosition(d)))}:function(a,b){if(b)while(b=b.parentNode)if(b===a)return!0;return!1},B=b?function(a,b){if(a===b)return l=!0,0;var d=!a.compareDocumentPosition-!b.compareDocumentPosition;return d?d:(d=(a.ownerDocument||a)===(b.ownerDocument||b)?a.compareDocumentPosition(b):1,1&d||!c.sortDetached&&b.compareDocumentPosition(a)===d?a===e||a.ownerDocument===v&&t(v,a)?-1:b===e||b.ownerDocument===v&&t(v,b)?1:k?K.call(k,a)-K.call(k,b):0:4&d?-1:1)}:function(a,b){if(a===b)return l=!0,0;var c,d=0,f=a.parentNode,g=b.parentNode,h=[a],i=[b];if(!f||!g)return a===e?-1:b===e?1:f?-1:g?1:k?K.call(k,a)-K.call(k,b):0;if(f===g)return kb(a,b);c=a;while(c=c.parentNode)h.unshift(c);c=b;while(c=c.parentNode)i.unshift(c);while(h[d]=
==i[d])d++;return d?kb(h[d],i[d]):h[d]===v?-1:i[d]===v?1:0},e):n},fb.matches=function(a,b){return fb(a,null,null,b)},fb.matchesSelector=function(a,b){if((a.ownerDocument||a)!==n&&m(a),b=b.replace(U,"='$1']"),!(!c.matchesSelector||!p||r&&r.test(b)||q&&q.test(b)))try{var d=s.call(a,b);if(d||c.disconnectedMatch||a.document&&11!==a.document.nodeType)return d}catch(e){}return fb(b,n,null,[a]).length>0},fb.contains=function(a,b){return(a.ownerDocument||a)!==n&&m(a),t(a,b)},fb.attr=function(a,b){(a.ownerDocument||a)!==n&&m(a);var e=d.attrHandle[b.toLowerCase()],f=e&&E.call(d.attrHandle,b.toLowerCase())?e(a,b,!p):void 0;return void 0!==f?f:c.attributes||!p?a.getAttribute(b):(f=a.getAttributeNode(b))&&f.specified?f.value:null},fb.error=function(a){throw new Error("Syntax error, unrecognized expression: "+a)},fb.uniqueSort=function(a){var b,d=[],e=0,f=0;if(l=!c.detectDuplicates,k=!c.sortStable&&a.slice(0),a.sort(B),l){while(b=a[f++])b===a[f]&&(e=d.push(f));while(e--)a.splice(d[e],1)}return k=
null,a},e=fb.getText=function(a){var b,c="",d=0,f=a.nodeType;if(f){if(1===f||9===f||11===f){if("string"==typeof a.textContent)return a.textContent;for(a=a.firstChild;a;a=a.nextSibling)c+=e(a)}else if(3===f||4===f)return a.nodeValue}else while(b=a[d++])c+=e(b);return c},d=fb.selectors={cacheLength:50,createPseudo:hb,match:X,attrHandle:{},find:{},relative:{">":{dir:"parentNode",first:!0}," ":{dir:"parentNode"},"+":{dir:"previousSibling",first:!0},"~":{dir:"previousSibling"}},preFilter:{ATTR:function(a){return a[1]=a[1].replace(cb,db),a[3]=(a[3]||a[4]||a[5]||"").replace(cb,db),"~="===a[2]&&(a[3]=" "+a[3]+" "),a.slice(0,4)},CHILD:function(a){return a[1]=a[1].toLowerCase(),"nth"===a[1].slice(0,3)?(a[3]||fb.error(a[0]),a[4]=+(a[4]?a[5]+(a[6]||1):2*("even"===a[3]||"odd"===a[3])),a[5]=+(a[7]+a[8]||"odd"===a[3])):a[3]&&fb.error(a[0]),a},PSEUDO:function(a){var b,c=!a[6]&&a[2];return X.CHILD.test(a[0])?null:(a[3]?a[2]=a[4]||a[5]||"":c&&V.test(c)&&(b=g(c,!0))&&(b=c.indexOf(")",c.length-b)-c.len
gth)&&(a[0]=a[0].slice(0,b),a[2]=c.slice(0,b)),a.slice(0,3))}},filter:{TAG:function(a){var b=a.replace(cb,db).toLowerCase();return"*"===a?function(){return!0}:function(a){return a.nodeName&&a.nodeName.toLowerCase()===b}},CLASS:function(a){var b=y[a+" "];return b||(b=new RegExp("(^|"+M+")"+a+"("+M+"|$)"))&&y(a,function(a){return b.test("string"==typeof a.className&&a.className||typeof a.getAttribute!==C&&a.getAttribute("class")||"")})},ATTR:function(a,b,c){return function(d){var e=fb.attr(d,a);return null==e?"!="===b:b?(e+="","="===b?e===c:"!="===b?e!==c:"^="===b?c&&0===e.indexOf(c):"*="===b?c&&e.indexOf(c)>-1:"$="===b?c&&e.slice(-c.length)===c:"~="===b?(" "+e+" ").indexOf(c)>-1:"|="===b?e===c||e.slice(0,c.length+1)===c+"-":!1):!0}},CHILD:function(a,b,c,d,e){var f="nth"!==a.slice(0,3),g="last"!==a.slice(-4),h="of-type"===b;return 1===d&&0===e?function(a){return!!a.parentNode}:function(b,c,i){var j,k,l,m,n,o,p=f!==g?"nextSibling":"previousSibling",q=b.parentNode,r=h&&b.nodeName.toLowe
rCase(),s=!i&&!h;if(q){if(f){while(p){l=b;while(l=l[p])if(h?l.nodeName.toLowerCase()===r:1===l.nodeType)return!1;o=p="only"===a&&!o&&"nextSibling"}return!0}if(o=[g?q.firstChild:q.lastChild],g&&s){k=q[u]||(q[u]={}),j=k[a]||[],n=j[0]===w&&j[1],m=j[0]===w&&j[2],l=n&&q.childNodes[n];while(l=++n&&l&&l[p]||(m=n=0)||o.pop())if(1===l.nodeType&&++m&&l===b){k[a]=[w,n,m];break}}else if(s&&(j=(b[u]||(b[u]={}))[a])&&j[0]===w)m=j[1];else while(l=++n&&l&&l[p]||(m=n=0)||o.pop())if((h?l.nodeName.toLowerCase()===r:1===l.nodeType)&&++m&&(s&&((l[u]||(l[u]={}))[a]=[w,m]),l===b))break;return m-=e,m===d||m%d===0&&m/d>=0}}},PSEUDO:function(a,b){var c,e=d.pseudos[a]||d.setFilters[a.toLowerCase()]||fb.error("unsupported pseudo: "+a);return e[u]?e(b):e.length>1?(c=[a,a,"",b],d.setFilters.hasOwnProperty(a.toLowerCase())?hb(function(a,c){var d,f=e(a,b),g=f.length;while(g--)d=K.call(a,f[g]),a[d]=!(c[d]=f[g])}):function(a){return e(a,0,c)}):e}},pseudos:{not:hb(function(a){var b=[],c=[],d=h(a.replace(R,"$1"));retu
rn d[u]?hb(function(a,b,c,e){var f,g=d(a,null,e,[]),h=a.length;while(h--)(f=g[h])&&(a[h]=!(b[h]=f))}):function(a,e,f){return b[0]=a,d(b,null,f,c),!c.pop()}}),has:hb(function(a){return function(b){return fb(a,b).length>0}}),contains:hb(function(a){return function(b){return(b.textContent||b.innerText||e(b)).indexOf(a)>-1}}),lang:hb(function(a){return W.test(a||"")||fb.error("unsupported lang: "+a),a=a.replace(cb,db).toLowerCase(),function(b){var c;do if(c=p?b.lang:b.getAttribute("xml:lang")||b.getAttribute("lang"))return c=c.toLowerCase(),c===a||0===c.indexOf(a+"-");while((b=b.parentNode)&&1===b.nodeType);return!1}}),target:function(b){var c=a.location&&a.location.hash;return c&&c.slice(1)===b.id},root:function(a){return a===o},focus:function(a){return a===n.activeElement&&(!n.hasFocus||n.hasFocus())&&!!(a.type||a.href||~a.tabIndex)},enabled:function(a){return a.disabled===!1},disabled:function(a){return a.disabled===!0},checked:function(a){var b=a.nodeName.toLowerCase();return"input"
===b&&!!a.checked||"option"===b&&!!a.selected},selected:function(a){return a.parentNode&&a.parentNode.selectedIndex,a.selected===!0},empty:function(a){for(a=a.firstChild;a;a=a.nextSibling)if(a.nodeType<6)return!1;return!0},parent:function(a){return!d.pseudos.empty(a)},header:function(a){return Z.test(a.nodeName)},input:function(a){return Y.test(a.nodeName)},button:function(a){var b=a.nodeName.toLowerCase();return"input"===b&&"button"===a.type||"button"===b},text:function(a){var b;return"input"===a.nodeName.toLowerCase()&&"text"===a.type&&(null==(b=a.getAttribute("type"))||"text"===b.toLowerCase())},first:nb(function(){return[0]}),last:nb(function(a,b){return[b-1]}),eq:nb(function(a,b,c){return[0>c?c+b:c]}),even:nb(function(a,b){for(var c=0;b>c;c+=2)a.push(c);return a}),odd:nb(function(a,b){for(var c=1;b>c;c+=2)a.push(c);return a}),lt:nb(function(a,b,c){for(var d=0>c?c+b:c;--d>=0;)a.push(d);return a}),gt:nb(function(a,b,c){for(var d=0>c?c+b:c;++d<b;)a.push(d);return a})}},d.pseudos.n
th=d.pseudos.eq;for(b in{radio:!0,checkbox:!0,file:!0,password:!0,image:!0})d.pseudos[b]=lb(b);for(b in{submit:!0,reset:!0})d.pseudos[b]=mb(b);function pb(){}pb.prototype=d.filters=d.pseudos,d.setFilters=new pb,g=fb.tokenize=function(a,b){var c,e,f,g,h,i,j,k=z[a+" "];if(k)return b?0:k.slice(0);h=a,i=[],j=d.preFilter;while(h){(!c||(e=S.exec(h)))&&(e&&(h=h.slice(e[0].length)||h),i.push(f=[])),c=!1,(e=T.exec(h))&&(c=e.shift(),f.push({value:c,type:e[0].replace(R," ")}),h=h.slice(c.length));for(g in d.filter)!(e=X[g].exec(h))||j[g]&&!(e=j[g](e))||(c=e.shift(),f.push({value:c,type:g,matches:e}),h=h.slice(c.length));if(!c)break}return b?h.length:h?fb.error(a):z(a,i).slice(0)};function qb(a){for(var b=0,c=a.length,d="";c>b;b++)d+=a[b].value;return d}function rb(a,b,c){var d=b.dir,e=c&&"parentNode"===d,f=x++;return b.first?function(b,c,f){while(b=b[d])if(1===b.nodeType||e)return a(b,c,f)}:function(b,c,g){var h,i,j=[w,f];if(g){while(b=b[d])if((1===b.nodeType||e)&&a(b,c,g))return!0}else while(
b=b[d])if(1===b.nodeType||e){if(i=b[u]||(b[u]={}),(h=i[d])&&h[0]===w&&h[1]===f)return j[2]=h[2];if(i[d]=j,j[2]=a(b,c,g))return!0}}}function sb(a){return a.length>1?function(b,c,d){var e=a.length;while(e--)if(!a[e](b,c,d))return!1;return!0}:a[0]}function tb(a,b,c){for(var d=0,e=b.length;e>d;d++)fb(a,b[d],c);return c}function ub(a,b,c,d,e){for(var f,g=[],h=0,i=a.length,j=null!=b;i>h;h++)(f=a[h])&&(!c||c(f,d,e))&&(g.push(f),j&&b.push(h));return g}function vb(a,b,c,d,e,f){return d&&!d[u]&&(d=vb(d)),e&&!e[u]&&(e=vb(e,f)),hb(function(f,g,h,i){var j,k,l,m=[],n=[],o=g.length,p=f||tb(b||"*",h.nodeType?[h]:h,[]),q=!a||!f&&b?p:ub(p,m,a,h,i),r=c?e||(f?a:o||d)?[]:g:q;if(c&&c(q,r,h,i),d){j=ub(r,n),d(j,[],h,i),k=j.length;while(k--)(l=j[k])&&(r[n[k]]=!(q[n[k]]=l))}if(f){if(e||a){if(e){j=[],k=r.length;while(k--)(l=r[k])&&j.push(q[k]=l);e(null,r=[],j,i)}k=r.length;while(k--)(l=r[k])&&(j=e?K.call(f,l):m[k])>-1&&(f[j]=!(g[j]=l))}}else r=ub(r===g?r.splice(o,r.length):r),e?e(null,g,r,i):I.apply(g,r)})}fu
nction wb(a){for(var b,c,e,f=a.length,g=d.relative[a[0].type],h=g||d.relative[" "],i=g?1:0,k=rb(function(a){return a===b},h,!0),l=rb(function(a){return K.call(b,a)>-1},h,!0),m=[function(a,c,d){return!g&&(d||c!==j)||((b=c).nodeType?k(a,c,d):l(a,c,d))}];f>i;i++)if(c=d.relative[a[i].type])m=[rb(sb(m),c)];else{if(c=d.filter[a[i].type].apply(null,a[i].matches),c[u]){for(e=++i;f>e;e++)if(d.relative[a[e].type])break;return vb(i>1&&sb(m),i>1&&qb(a.slice(0,i-1).concat({value:" "===a[i-2].type?"*":""})).replace(R,"$1"),c,e>i&&wb(a.slice(i,e)),f>e&&wb(a=a.slice(e)),f>e&&qb(a))}m.push(c)}return sb(m)}function xb(a,b){var c=b.length>0,e=a.length>0,f=function(f,g,h,i,k){var l,m,o,p=0,q="0",r=f&&[],s=[],t=j,u=f||e&&d.find.TAG("*",k),v=w+=null==t?1:Math.random()||.1,x=u.length;for(k&&(j=g!==n&&g);q!==x&&null!=(l=u[q]);q++){if(e&&l){m=0;while(o=a[m++])if(o(l,g,h)){i.push(l);break}k&&(w=v)}c&&((l=!o&&l)&&p--,f&&r.push(l))}if(p+=q,c&&q!==p){m=0;while(o=b[m++])o(r,s,g,h);if(f){if(p>0)while(q--)r[q]||s[
q]||(s[q]=G.call(i));s=ub(s)}I.apply(i,s),k&&!f&&s.length>0&&p+b.length>1&&fb.uniqueSort(i)}return k&&(w=v,j=t),r};return c?hb(f):f}return h=fb.compile=function(a,b){var c,d=[],e=[],f=A[a+" "];if(!f){b||(b=g(a)),c=b.length;while(c--)f=wb(b[c]),f[u]?d.push(f):e.push(f);f=A(a,xb(e,d)),f.selector=a}return f},i=fb.select=function(a,b,e,f){var i,j,k,l,m,n="function"==typeof a&&a,o=!f&&g(a=n.selector||a);if(e=e||[],1===o.length){if(j=o[0]=o[0].slice(0),j.length>2&&"ID"===(k=j[0]).type&&c.getById&&9===b.nodeType&&p&&d.relative[j[1].type]){if(b=(d.find.ID(k.matches[0].replace(cb,db),b)||[])[0],!b)return e;n&&(b=b.parentNode),a=a.slice(j.shift().value.length)}i=X.needsContext.test(a)?0:j.length;while(i--){if(k=j[i],d.relative[l=k.type])break;if((m=d.find[l])&&(f=m(k.matches[0].replace(cb,db),ab.test(j[0].type)&&ob(b.parentNode)||b))){if(j.splice(i,1),a=f.length&&qb(j),!a)return I.apply(e,f),e;break}}}return(n||h(a,o))(f,b,!p,e,ab.test(a)&&ob(b.parentNode)||b),e},c.sortStable=u.split("").sort
(B).join("")===u,c.detectDuplicates=!!l,m(),c.sortDetached=ib(function(a){return 1&a.compareDocumentPosition(n.createElement("div"))}),ib(function(a){return a.innerHTML="<a href='#'></a>","#"===a.firstChild.getAttribute("href")})||jb("type|href|height|width",function(a,b,c){return c?void 0:a.getAttribute(b,"type"===b.toLowerCase()?1:2)}),c.attributes&&ib(function(a){return a.innerHTML="<input/>",a.firstChild.setAttribute("value",""),""===a.firstChild.getAttribute("value")})||jb("value",function(a,b,c){return c||"input"!==a.nodeName.toLowerCase()?void 0:a.defaultValue}),ib(function(a){return null==a.getAttribute("disabled")})||jb(L,function(a,b,c){var d;return c?void 0:a[b]===!0?b.toLowerCase():(d=a.getAttributeNode(b))&&d.specified?d.value:null}),fb}(a);m.find=s,m.expr=s.selectors,m.expr[":"]=m.expr.pseudos,m.unique=s.uniqueSort,m.text=s.getText,m.isXMLDoc=s.isXML,m.contains=s.contains;var t=m.expr.match.needsContext,u=/^<(\w+)\s*\/?>(?:<\/\1>|)$/,v=/^.[^:#\[\.,]*$/;function w(a,b,c
){if(m.isFunction(b))return m.grep(a,function(a,d){return!!b.call(a,d,a)!==c});if(b.nodeType)return m.grep(a,function(a){return a===b!==c});if("string"==typeof b){if(v.test(b))return m.filter(b,a,c);b=m.filter(b,a)}return m.grep(a,function(a){return m.inArray(a,b)>=0!==c})}m.filter=function(a,b,c){var d=b[0];return c&&(a=":not("+a+")"),1===b.length&&1===d.nodeType?m.find.matchesSelector(d,a)?[d]:[]:m.find.matches(a,m.grep(b,function(a){return 1===a.nodeType}))},m.fn.extend({find:function(a){var b,c=[],d=this,e=d.length;if("string"!=typeof a)return this.pushStack(m(a).filter(function(){for(b=0;e>b;b++)if(m.contains(d[b],this))return!0}));for(b=0;e>b;b++)m.find(a,d[b],c);return c=this.pushStack(e>1?m.unique(c):c),c.selector=this.selector?this.selector+" "+a:a,c},filter:function(a){return this.pushStack(w(this,a||[],!1))},not:function(a){return this.pushStack(w(this,a||[],!0))},is:function(a){return!!w(this,"string"==typeof a&&t.test(a)?m(a):a||[],!1).length}});var x,y=a.document,z=/^(
?:\s*(<[\w\W]+>)[^>]*|#([\w-]*))$/,A=m.fn.init=function(a,b){var c,d;if(!a)return this;if("string"==typeof a){if(c="<"===a.charAt(0)&&">"===a.charAt(a.length-1)&&a.length>=3?[null,a,null]:z.exec(a),!c||!c[1]&&b)return!b||b.jquery?(b||x).find(a):this.constructor(b).find(a);if(c[1]){if(b=b instanceof m?b[0]:b,m.merge(this,m.parseHTML(c[1],b&&b.nodeType?b.ownerDocument||b:y,!0)),u.test(c[1])&&m.isPlainObject(b))for(c in b)m.isFunction(this[c])?this[c](b[c]):this.attr(c,b[c]);return this}if(d=y.getElementById(c[2]),d&&d.parentNode){if(d.id!==c[2])return x.find(a);this.length=1,this[0]=d}return this.context=y,this.selector=a,this}return a.nodeType?(this.context=this[0]=a,this.length=1,this):m.isFunction(a)?"undefined"!=typeof x.ready?x.ready(a):a(m):(void 0!==a.selector&&(this.selector=a.selector,this.context=a.context),m.makeArray(a,this))};A.prototype=m.fn,x=m(y);var B=/^(?:parents|prev(?:Until|All))/,C={children:!0,contents:!0,next:!0,prev:!0};m.extend({dir:function(a,b,c){var d=[],e=
a[b];while(e&&9!==e.nodeType&&(void 0===c||1!==e.nodeType||!m(e).is(c)))1===e.nodeType&&d.push(e),e=e[b];return d},sibling:function(a,b){for(var c=[];a;a=a.nextSibling)1===a.nodeType&&a!==b&&c.push(a);return c}}),m.fn.extend({has:function(a){var b,c=m(a,this),d=c.length;return this.filter(function(){for(b=0;d>b;b++)if(m.contains(this,c[b]))return!0})},closest:function(a,b){for(var c,d=0,e=this.length,f=[],g=t.test(a)||"string"!=typeof a?m(a,b||this.context):0;e>d;d++)for(c=this[d];c&&c!==b;c=c.parentNode)if(c.nodeType<11&&(g?g.index(c)>-1:1===c.nodeType&&m.find.matchesSelector(c,a))){f.push(c);break}return this.pushStack(f.length>1?m.unique(f):f)},index:function(a){return a?"string"==typeof a?m.inArray(this[0],m(a)):m.inArray(a.jquery?a[0]:a,this):this[0]&&this[0].parentNode?this.first().prevAll().length:-1},add:function(a,b){return this.pushStack(m.unique(m.merge(this.get(),m(a,b))))},addBack:function(a){return this.add(null==a?this.prevObject:this.prevObject.filter(a))}});function
D(a,b){do a=a[b];while(a&&1!==a.nodeType);return a}m.each({parent:function(a){var b=a.parentNode;return b&&11!==b.nodeType?b:null},parents:function(a){return m.dir(a,"parentNode")},parentsUntil:function(a,b,c){return m.dir(a,"parentNode",c)},next:function(a){return D(a,"nextSibling")},prev:function(a){return D(a,"previousSibling")},nextAll:function(a){return m.dir(a,"nextSibling")},prevAll:function(a){return m.dir(a,"previousSibling")},nextUntil:function(a,b,c){return m.dir(a,"nextSibling",c)},prevUntil:function(a,b,c){return m.dir(a,"previousSibling",c)},siblings:function(a){return m.sibling((a.parentNode||{}).firstChild,a)},children:function(a){return m.sibling(a.firstChild)},contents:function(a){return m.nodeName(a,"iframe")?a.contentDocument||a.contentWindow.document:m.merge([],a.childNodes)}},function(a,b){m.fn[a]=function(c,d){var e=m.map(this,b,c);return"Until"!==a.slice(-5)&&(d=c),d&&"string"==typeof d&&(e=m.filter(d,e)),this.length>1&&(C[a]||(e=m.unique(e)),B.test(a)&&(e=e
.reverse())),this.pushStack(e)}});var E=/\S+/g,F={};function G(a){var b=F[a]={};return m.each(a.match(E)||[],function(a,c){b[c]=!0}),b}m.Callbacks=function(a){a="string"==typeof a?F[a]||G(a):m.extend({},a);var b,c,d,e,f,g,h=[],i=!a.once&&[],j=function(l){for(c=a.memory&&l,d=!0,f=g||0,g=0,e=h.length,b=!0;h&&e>f;f++)if(h[f].apply(l[0],l[1])===!1&&a.stopOnFalse){c=!1;break}b=!1,h&&(i?i.length&&j(i.shift()):c?h=[]:k.disable())},k={add:function(){if(h){var d=h.length;!function f(b){m.each(b,function(b,c){var d=m.type(c);"function"===d?a.unique&&k.has(c)||h.push(c):c&&c.length&&"string"!==d&&f(c)})}(arguments),b?e=h.length:c&&(g=d,j(c))}return this},remove:function(){return h&&m.each(arguments,function(a,c){var d;while((d=m.inArray(c,h,d))>-1)h.splice(d,1),b&&(e>=d&&e--,f>=d&&f--)}),this},has:function(a){return a?m.inArray(a,h)>-1:!(!h||!h.length)},empty:function(){return h=[],e=0,this},disable:function(){return h=i=c=void 0,this},disabled:function(){return!h},lock:function(){return i=voi
d 0,c||k.disable(),this},locked:function(){return!i},fireWith:function(a,c){return!h||d&&!i||(c=c||[],c=[a,c.slice?c.slice():c],b?i.push(c):j(c)),this},fire:function(){return k.fireWith(this,arguments),this},fired:function(){return!!d}};return k},m.extend({Deferred:function(a){var b=[["resolve","done",m.Callbacks("once memory"),"resolved"],["reject","fail",m.Callbacks("once memory"),"rejected"],["notify","progress",m.Callbacks("memory")]],c="pending",d={state:function(){return c},always:function(){return e.done(arguments).fail(arguments),this},then:function(){var a=arguments;return m.Deferred(function(c){m.each(b,function(b,f){var g=m.isFunction(a[b])&&a[b];e[f[1]](function(){var a=g&&g.apply(this,arguments);a&&m.isFunction(a.promise)?a.promise().done(c.resolve).fail(c.reject).progress(c.notify):c[f[0]+"With"](this===d?c.promise():this,g?[a]:arguments)})}),a=null}).promise()},promise:function(a){return null!=a?m.extend(a,d):d}},e={};return d.pipe=d.then,m.each(b,function(a,f){var g=
f[2],h=f[3];d[f[1]]=g.add,h&&g.add(function(){c=h},b[1^a][2].disable,b[2][2].lock),e[f[0]]=function(){return e[f[0]+"With"](this===e?d:this,arguments),this},e[f[0]+"With"]=g.fireWith}),d.promise(e),a&&a.call(e,e),e},when:function(a){var b=0,c=d.call(arguments),e=c.length,f=1!==e||a&&m.isFunction(a.promise)?e:0,g=1===f?a:m.Deferred(),h=function(a,b,c){return function(e){b[a]=this,c[a]=arguments.length>1?d.call(arguments):e,c===i?g.notifyWith(b,c):--f||g.resolveWith(b,c)}},i,j,k;if(e>1)for(i=new Array(e),j=new Array(e),k=new Array(e);e>b;b++)c[b]&&m.isFunction(c[b].promise)?c[b].promise().done(h(b,k,c)).fail(g.reject).progress(h(b,j,i)):--f;return f||g.resolveWith(k,c),g.promise()}});var H;m.fn.ready=function(a){return m.ready.promise().done(a),this},m.extend({isReady:!1,readyWait:1,holdReady:function(a){a?m.readyWait++:m.ready(!0)},ready:function(a){if(a===!0?!--m.readyWait:!m.isReady){if(!y.body)return setTimeout(m.ready);m.isReady=!0,a!==!0&&--m.readyWait>0||(H.resolveWith(y,[m]),m
.fn.triggerHandler&&(m(y).triggerHandler("ready"),m(y).off("ready")))}}});function I(){y.addEventListener?(y.removeEventListener("DOMContentLoaded",J,!1),a.removeEventListener("load",J,!1)):(y.detachEvent("onreadystatechange",J),a.detachEvent("onload",J))}function J(){(y.addEventListener||"load"===event.type||"complete"===y.readyState)&&(I(),m.ready())}m.ready.promise=function(b){if(!H)if(H=m.Deferred(),"complete"===y.readyState)setTimeout(m.ready);else if(y.addEventListener)y.addEventListener("DOMContentLoaded",J,!1),a.addEventListener("load",J,!1);else{y.attachEvent("onreadystatechange",J),a.attachEvent("onload",J);var c=!1;try{c=null==a.frameElement&&y.documentElement}catch(d){}c&&c.doScroll&&!function e(){if(!m.isReady){try{c.doScroll("left")}catch(a){return setTimeout(e,50)}I(),m.ready()}}()}return H.promise(b)};var K="undefined",L;for(L in m(k))break;k.ownLast="0"!==L,k.inlineBlockNeedsLayout=!1,m(function(){var a,b,c,d;c=y.getElementsByTagName("body")[0],c&&c.style&&(b=y.crea
teElement("div"),d=y.createElement("div"),d.style.cssText="position:absolute;border:0;width:0;height:0;top:0;left:-9999px",c.appendChild(d).appendChild(b),typeof b.style.zoom!==K&&(b.style.cssText="display:inline;margin:0;border:0;padding:1px;width:1px;zoom:1",k.inlineBlockNeedsLayout=a=3===b.offsetWidth,a&&(c.style.zoom=1)),c.removeChild(d))}),function(){var a=y.createElement("div");if(null==k.deleteExpando){k.deleteExpando=!0;try{delete a.test}catch(b){k.deleteExpando=!1}}a=null}(),m.acceptData=function(a){var b=m.noData[(a.nodeName+" ").toLowerCase()],c=+a.nodeType||1;return 1!==c&&9!==c?!1:!b||b!==!0&&a.getAttribute("classid")===b};var M=/^(?:\{[\w\W]*\}|\[[\w\W]*\])$/,N=/([A-Z])/g;function O(a,b,c){if(void 0===c&&1===a.nodeType){var d="data-"+b.replace(N,"-$1").toLowerCase();if(c=a.getAttribute(d),"string"==typeof c){try{c="true"===c?!0:"false"===c?!1:"null"===c?null:+c+""===c?+c:M.test(c)?m.parseJSON(c):c}catch(e){}m.data(a,b,c)}else c=void 0}return c}function P(a){var b;for(b
in a)if(("data"!==b||!m.isEmptyObject(a[b]))&&"toJSON"!==b)return!1;return!0}function Q(a,b,d,e){if(m.acceptData(a)){var f,g,h=m.expando,i=a.nodeType,j=i?m.cache:a,k=i?a[h]:a[h]&&h;
+if(k&&j[k]&&(e||j[k].data)||void 0!==d||"string"!=typeof b)return k||(k=i?a[h]=c.pop()||m.guid++:h),j[k]||(j[k]=i?{}:{toJSON:m.noop}),("object"==typeof b||"function"==typeof b)&&(e?j[k]=m.extend(j[k],b):j[k].data=m.extend(j[k].data,b)),g=j[k],e||(g.data||(g.data={}),g=g.data),void 0!==d&&(g[m.camelCase(b)]=d),"string"==typeof b?(f=g[b],null==f&&(f=g[m.camelCase(b)])):f=g,f}}function R(a,b,c){if(m.acceptData(a)){var d,e,f=a.nodeType,g=f?m.cache:a,h=f?a[m.expando]:m.expando;if(g[h]){if(b&&(d=c?g[h]:g[h].data)){m.isArray(b)?b=b.concat(m.map(b,m.camelCase)):b in d?b=[b]:(b=m.camelCase(b),b=b in d?[b]:b.split(" ")),e=b.length;while(e--)delete d[b[e]];if(c?!P(d):!m.isEmptyObject(d))return}(c||(delete g[h].data,P(g[h])))&&(f?m.cleanData([a],!0):k.deleteExpando||g!=g.window?delete g[h]:g[h]=null)}}}m.extend({cache:{},noData:{"applet ":!0,"embed ":!0,"object ":"clsid:D27CDB6E-AE6D-11cf-96B8-444553540000"},hasData:function(a){return a=a.nodeType?m.cache[a[m.expando]]:a[m.expando],!!a&&!P(a)},
data:function(a,b,c){return Q(a,b,c)},removeData:function(a,b){return R(a,b)},_data:function(a,b,c){return Q(a,b,c,!0)},_removeData:function(a,b){return R(a,b,!0)}}),m.fn.extend({data:function(a,b){var c,d,e,f=this[0],g=f&&f.attributes;if(void 0===a){if(this.length&&(e=m.data(f),1===f.nodeType&&!m._data(f,"parsedAttrs"))){c=g.length;while(c--)g[c]&&(d=g[c].name,0===d.indexOf("data-")&&(d=m.camelCase(d.slice(5)),O(f,d,e[d])));m._data(f,"parsedAttrs",!0)}return e}return"object"==typeof a?this.each(function(){m.data(this,a)}):arguments.length>1?this.each(function(){m.data(this,a,b)}):f?O(f,a,m.data(f,a)):void 0},removeData:function(a){return this.each(function(){m.removeData(this,a)})}}),m.extend({queue:function(a,b,c){var d;return a?(b=(b||"fx")+"queue",d=m._data(a,b),c&&(!d||m.isArray(c)?d=m._data(a,b,m.makeArray(c)):d.push(c)),d||[]):void 0},dequeue:function(a,b){b=b||"fx";var c=m.queue(a,b),d=c.length,e=c.shift(),f=m._queueHooks(a,b),g=function(){m.dequeue(a,b)};"inprogress"===e&&(
e=c.shift(),d--),e&&("fx"===b&&c.unshift("inprogress"),delete f.stop,e.call(a,g,f)),!d&&f&&f.empty.fire()},_queueHooks:function(a,b){var c=b+"queueHooks";return m._data(a,c)||m._data(a,c,{empty:m.Callbacks("once memory").add(function(){m._removeData(a,b+"queue"),m._removeData(a,c)})})}}),m.fn.extend({queue:function(a,b){var c=2;return"string"!=typeof a&&(b=a,a="fx",c--),arguments.length<c?m.queue(this[0],a):void 0===b?this:this.each(function(){var c=m.queue(this,a,b);m._queueHooks(this,a),"fx"===a&&"inprogress"!==c[0]&&m.dequeue(this,a)})},dequeue:function(a){return this.each(function(){m.dequeue(this,a)})},clearQueue:function(a){return this.queue(a||"fx",[])},promise:function(a,b){var c,d=1,e=m.Deferred(),f=this,g=this.length,h=function(){--d||e.resolveWith(f,[f])};"string"!=typeof a&&(b=a,a=void 0),a=a||"fx";while(g--)c=m._data(f[g],a+"queueHooks"),c&&c.empty&&(d++,c.empty.add(h));return h(),e.promise(b)}});var S=/[+-]?(?:\d*\.|)\d+(?:[eE][+-]?\d+|)/.source,T=["Top","Right","Botto
m","Left"],U=function(a,b){return a=b||a,"none"===m.css(a,"display")||!m.contains(a.ownerDocument,a)},V=m.access=function(a,b,c,d,e,f,g){var h=0,i=a.length,j=null==c;if("object"===m.type(c)){e=!0;for(h in c)m.access(a,b,h,c[h],!0,f,g)}else if(void 0!==d&&(e=!0,m.isFunction(d)||(g=!0),j&&(g?(b.call(a,d),b=null):(j=b,b=function(a,b,c){return j.call(m(a),c)})),b))for(;i>h;h++)b(a[h],c,g?d:d.call(a[h],h,b(a[h],c)));return e?a:j?b.call(a):i?b(a[0],c):f},W=/^(?:checkbox|radio)$/i;!function(){var a=y.createElement("input"),b=y.createElement("div"),c=y.createDocumentFragment();if(b.innerHTML=" <link/><table></table><a href='/a'>a</a><input type='checkbox'/>",k.leadingWhitespace=3===b.firstChild.nodeType,k.tbody=!b.getElementsByTagName("tbody").length,k.htmlSerialize=!!b.getElementsByTagName("link").length,k.html5Clone="<:nav></:nav>"!==y.createElement("nav").cloneNode(!0).outerHTML,a.type="checkbox",a.checked=!0,c.appendChild(a),k.appendChecked=a.checked,b.innerHTML="<textarea>x</textarea>
",k.noCloneChecked=!!b.cloneNode(!0).lastChild.defaultValue,c.appendChild(b),b.innerHTML="<input type='radio' checked='checked' name='t'/>",k.checkClone=b.cloneNode(!0).cloneNode(!0).lastChild.checked,k.noCloneEvent=!0,b.attachEvent&&(b.attachEvent("onclick",function(){k.noCloneEvent=!1}),b.cloneNode(!0).click()),null==k.deleteExpando){k.deleteExpando=!0;try{delete b.test}catch(d){k.deleteExpando=!1}}}(),function(){var b,c,d=y.createElement("div");for(b in{submit:!0,change:!0,focusin:!0})c="on"+b,(k[b+"Bubbles"]=c in a)||(d.setAttribute(c,"t"),k[b+"Bubbles"]=d.attributes[c].expando===!1);d=null}();var X=/^(?:input|select|textarea)$/i,Y=/^key/,Z=/^(?:mouse|pointer|contextmenu)|click/,$=/^(?:focusinfocus|focusoutblur)$/,_=/^([^.]*)(?:\.(.+)|)$/;function ab(){return!0}function bb(){return!1}function cb(){try{return y.activeElement}catch(a){}}m.event={global:{},add:function(a,b,c,d,e){var f,g,h,i,j,k,l,n,o,p,q,r=m._data(a);if(r){c.handler&&(i=c,c=i.handler,e=i.selector),c.guid||(c.guid=
m.guid++),(g=r.events)||(g=r.events={}),(k=r.handle)||(k=r.handle=function(a){return typeof m===K||a&&m.event.triggered===a.type?void 0:m.event.dispatch.apply(k.elem,arguments)},k.elem=a),b=(b||"").match(E)||[""],h=b.length;while(h--)f=_.exec(b[h])||[],o=q=f[1],p=(f[2]||"").split(".").sort(),o&&(j=m.event.special[o]||{},o=(e?j.delegateType:j.bindType)||o,j=m.event.special[o]||{},l=m.extend({type:o,origType:q,data:d,handler:c,guid:c.guid,selector:e,needsContext:e&&m.expr.match.needsContext.test(e),namespace:p.join(".")},i),(n=g[o])||(n=g[o]=[],n.delegateCount=0,j.setup&&j.setup.call(a,d,p,k)!==!1||(a.addEventListener?a.addEventListener(o,k,!1):a.attachEvent&&a.attachEvent("on"+o,k))),j.add&&(j.add.call(a,l),l.handler.guid||(l.handler.guid=c.guid)),e?n.splice(n.delegateCount++,0,l):n.push(l),m.event.global[o]=!0);a=null}},remove:function(a,b,c,d,e){var f,g,h,i,j,k,l,n,o,p,q,r=m.hasData(a)&&m._data(a);if(r&&(k=r.events)){b=(b||"").match(E)||[""],j=b.length;while(j--)if(h=_.exec(b[j])||
[],o=q=h[1],p=(h[2]||"").split(".").sort(),o){l=m.event.special[o]||{},o=(d?l.delegateType:l.bindType)||o,n=k[o]||[],h=h[2]&&new RegExp("(^|\\.)"+p.join("\\.(?:.*\\.|)")+"(\\.|$)"),i=f=n.length;while(f--)g=n[f],!e&&q!==g.origType||c&&c.guid!==g.guid||h&&!h.test(g.namespace)||d&&d!==g.selector&&("**"!==d||!g.selector)||(n.splice(f,1),g.selector&&n.delegateCount--,l.remove&&l.remove.call(a,g));i&&!n.length&&(l.teardown&&l.teardown.call(a,p,r.handle)!==!1||m.removeEvent(a,o,r.handle),delete k[o])}else for(o in k)m.event.remove(a,o+b[j],c,d,!0);m.isEmptyObject(k)&&(delete r.handle,m._removeData(a,"events"))}},trigger:function(b,c,d,e){var f,g,h,i,k,l,n,o=[d||y],p=j.call(b,"type")?b.type:b,q=j.call(b,"namespace")?b.namespace.split("."):[];if(h=l=d=d||y,3!==d.nodeType&&8!==d.nodeType&&!$.test(p+m.event.triggered)&&(p.indexOf(".")>=0&&(q=p.split("."),p=q.shift(),q.sort()),g=p.indexOf(":")<0&&"on"+p,b=b[m.expando]?b:new m.Event(p,"object"==typeof b&&b),b.isTrigger=e?2:3,b.namespace=q.join("
."),b.namespace_re=b.namespace?new RegExp("(^|\\.)"+q.join("\\.(?:.*\\.|)")+"(\\.|$)"):null,b.result=void 0,b.target||(b.target=d),c=null==c?[b]:m.makeArray(c,[b]),k=m.event.special[p]||{},e||!k.trigger||k.trigger.apply(d,c)!==!1)){if(!e&&!k.noBubble&&!m.isWindow(d)){for(i=k.delegateType||p,$.test(i+p)||(h=h.parentNode);h;h=h.parentNode)o.push(h),l=h;l===(d.ownerDocument||y)&&o.push(l.defaultView||l.parentWindow||a)}n=0;while((h=o[n++])&&!b.isPropagationStopped())b.type=n>1?i:k.bindType||p,f=(m._data(h,"events")||{})[b.type]&&m._data(h,"handle"),f&&f.apply(h,c),f=g&&h[g],f&&f.apply&&m.acceptData(h)&&(b.result=f.apply(h,c),b.result===!1&&b.preventDefault());if(b.type=p,!e&&!b.isDefaultPrevented()&&(!k._default||k._default.apply(o.pop(),c)===!1)&&m.acceptData(d)&&g&&d[p]&&!m.isWindow(d)){l=d[g],l&&(d[g]=null),m.event.triggered=p;try{d[p]()}catch(r){}m.event.triggered=void 0,l&&(d[g]=l)}return b.result}},dispatch:function(a){a=m.event.fix(a);var b,c,e,f,g,h=[],i=d.call(arguments),j=(m.
_data(this,"events")||{})[a.type]||[],k=m.event.special[a.type]||{};if(i[0]=a,a.delegateTarget=this,!k.preDispatch||k.preDispatch.call(this,a)!==!1){h=m.event.handlers.call(this,a,j),b=0;while((f=h[b++])&&!a.isPropagationStopped()){a.currentTarget=f.elem,g=0;while((e=f.handlers[g++])&&!a.isImmediatePropagationStopped())(!a.namespace_re||a.namespace_re.test(e.namespace))&&(a.handleObj=e,a.data=e.data,c=((m.event.special[e.origType]||{}).handle||e.handler).apply(f.elem,i),void 0!==c&&(a.result=c)===!1&&(a.preventDefault(),a.stopPropagation()))}return k.postDispatch&&k.postDispatch.call(this,a),a.result}},handlers:function(a,b){var c,d,e,f,g=[],h=b.delegateCount,i=a.target;if(h&&i.nodeType&&(!a.button||"click"!==a.type))for(;i!=this;i=i.parentNode||this)if(1===i.nodeType&&(i.disabled!==!0||"click"!==a.type)){for(e=[],f=0;h>f;f++)d=b[f],c=d.selector+" ",void 0===e[c]&&(e[c]=d.needsContext?m(c,this).index(i)>=0:m.find(c,this,null,[i]).length),e[c]&&e.push(d);e.length&&g.push({elem:i,hand
lers:e})}return h<b.length&&g.push({elem:this,handlers:b.slice(h)}),g},fix:function(a){if(a[m.expando])return a;var b,c,d,e=a.type,f=a,g=this.fixHooks[e];g||(this.fixHooks[e]=g=Z.test(e)?this.mouseHooks:Y.test(e)?this.keyHooks:{}),d=g.props?this.props.concat(g.props):this.props,a=new m.Event(f),b=d.length;while(b--)c=d[b],a[c]=f[c];return a.target||(a.target=f.srcElement||y),3===a.target.nodeType&&(a.target=a.target.parentNode),a.metaKey=!!a.metaKey,g.filter?g.filter(a,f):a},props:"altKey bubbles cancelable ctrlKey currentTarget eventPhase metaKey relatedTarget shiftKey target timeStamp view which".split(" "),fixHooks:{},keyHooks:{props:"char charCode key keyCode".split(" "),filter:function(a,b){return null==a.which&&(a.which=null!=b.charCode?b.charCode:b.keyCode),a}},mouseHooks:{props:"button buttons clientX clientY fromElement offsetX offsetY pageX pageY screenX screenY toElement".split(" "),filter:function(a,b){var c,d,e,f=b.button,g=b.fromElement;return null==a.pageX&&null!=b.cl
ientX&&(d=a.target.ownerDocument||y,e=d.documentElement,c=d.body,a.pageX=b.clientX+(e&&e.scrollLeft||c&&c.scrollLeft||0)-(e&&e.clientLeft||c&&c.clientLeft||0),a.pageY=b.clientY+(e&&e.scrollTop||c&&c.scrollTop||0)-(e&&e.clientTop||c&&c.clientTop||0)),!a.relatedTarget&&g&&(a.relatedTarget=g===a.target?b.toElement:g),a.which||void 0===f||(a.which=1&f?1:2&f?3:4&f?2:0),a}},special:{load:{noBubble:!0},focus:{trigger:function(){if(this!==cb()&&this.focus)try{return this.focus(),!1}catch(a){}},delegateType:"focusin"},blur:{trigger:function(){return this===cb()&&this.blur?(this.blur(),!1):void 0},delegateType:"focusout"},click:{trigger:function(){return m.nodeName(this,"input")&&"checkbox"===this.type&&this.click?(this.click(),!1):void 0},_default:function(a){return m.nodeName(a.target,"a")}},beforeunload:{postDispatch:function(a){void 0!==a.result&&a.originalEvent&&(a.originalEvent.returnValue=a.result)}}},simulate:function(a,b,c,d){var e=m.extend(new m.Event,c,{type:a,isSimulated:!0,origin
alEvent:{}});d?m.event.trigger(e,null,b):m.event.dispatch.call(b,e),e.isDefaultPrevented()&&c.preventDefault()}},m.removeEvent=y.removeEventListener?function(a,b,c){a.removeEventListener&&a.removeEventListener(b,c,!1)}:function(a,b,c){var d="on"+b;a.detachEvent&&(typeof a[d]===K&&(a[d]=null),a.detachEvent(d,c))},m.Event=function(a,b){return this instanceof m.Event?(a&&a.type?(this.originalEvent=a,this.type=a.type,this.isDefaultPrevented=a.defaultPrevented||void 0===a.defaultPrevented&&a.returnValue===!1?ab:bb):this.type=a,b&&m.extend(this,b),this.timeStamp=a&&a.timeStamp||m.now(),void(this[m.expando]=!0)):new m.Event(a,b)},m.Event.prototype={isDefaultPrevented:bb,isPropagationStopped:bb,isImmediatePropagationStopped:bb,preventDefault:function(){var a=this.originalEvent;this.isDefaultPrevented=ab,a&&(a.preventDefault?a.preventDefault():a.returnValue=!1)},stopPropagation:function(){var a=this.originalEvent;this.isPropagationStopped=ab,a&&(a.stopPropagation&&a.stopPropagation(),a.cance
lBubble=!0)},stopImmediatePropagation:function(){var a=this.originalEvent;this.isImmediatePropagationStopped=ab,a&&a.stopImmediatePropagation&&a.stopImmediatePropagation(),this.stopPropagation()}},m.each({mouseenter:"mouseover",mouseleave:"mouseout",pointerenter:"pointerover",pointerleave:"pointerout"},function(a,b){m.event.special[a]={delegateType:b,bindType:b,handle:function(a){var c,d=this,e=a.relatedTarget,f=a.handleObj;return(!e||e!==d&&!m.contains(d,e))&&(a.type=f.origType,c=f.handler.apply(this,arguments),a.type=b),c}}}),k.submitBubbles||(m.event.special.submit={setup:function(){return m.nodeName(this,"form")?!1:void m.event.add(this,"click._submit keypress._submit",function(a){var b=a.target,c=m.nodeName(b,"input")||m.nodeName(b,"button")?b.form:void 0;c&&!m._data(c,"submitBubbles")&&(m.event.add(c,"submit._submit",function(a){a._submit_bubble=!0}),m._data(c,"submitBubbles",!0))})},postDispatch:function(a){a._submit_bubble&&(delete a._submit_bubble,this.parentNode&&!a.isTrig
ger&&m.event.simulate("submit",this.parentNode,a,!0))},teardown:function(){return m.nodeName(this,"form")?!1:void m.event.remove(this,"._submit")}}),k.changeBubbles||(m.event.special.change={setup:function(){return X.test(this.nodeName)?(("checkbox"===this.type||"radio"===this.type)&&(m.event.add(this,"propertychange._change",function(a){"checked"===a.originalEvent.propertyName&&(this._just_changed=!0)}),m.event.add(this,"click._change",function(a){this._just_changed&&!a.isTrigger&&(this._just_changed=!1),m.event.simulate("change",this,a,!0)})),!1):void m.event.add(this,"beforeactivate._change",function(a){var b=a.target;X.test(b.nodeName)&&!m._data(b,"changeBubbles")&&(m.event.add(b,"change._change",function(a){!this.parentNode||a.isSimulated||a.isTrigger||m.event.simulate("change",this.parentNode,a,!0)}),m._data(b,"changeBubbles",!0))})},handle:function(a){var b=a.target;return this!==b||a.isSimulated||a.isTrigger||"radio"!==b.type&&"checkbox"!==b.type?a.handleObj.handler.apply(th
is,arguments):void 0},teardown:function(){return m.event.remove(this,"._change"),!X.test(this.nodeName)}}),k.focusinBubbles||m.each({focus:"focusin",blur:"focusout"},function(a,b){var c=function(a){m.event.simulate(b,a.target,m.event.fix(a),!0)};m.event.special[b]={setup:function(){var d=this.ownerDocument||this,e=m._data(d,b);e||d.addEventListener(a,c,!0),m._data(d,b,(e||0)+1)},teardown:function(){var d=this.ownerDocument||this,e=m._data(d,b)-1;e?m._data(d,b,e):(d.removeEventListener(a,c,!0),m._removeData(d,b))}}}),m.fn.extend({on:function(a,b,c,d,e){var f,g;if("object"==typeof a){"string"!=typeof b&&(c=c||b,b=void 0);for(f in a)this.on(f,b,c,a[f],e);return this}if(null==c&&null==d?(d=b,c=b=void 0):null==d&&("string"==typeof b?(d=c,c=void 0):(d=c,c=b,b=void 0)),d===!1)d=bb;else if(!d)return this;return 1===e&&(g=d,d=function(a){return m().off(a),g.apply(this,arguments)},d.guid=g.guid||(g.guid=m.guid++)),this.each(function(){m.event.add(this,a,d,c,b)})},one:function(a,b,c,d){return
this.on(a,b,c,d,1)},off:function(a,b,c){var d,e;if(a&&a.preventDefault&&a.handleObj)return d=a.handleObj,m(a.delegateTarget).off(d.namespace?d.origType+"."+d.namespace:d.origType,d.selector,d.handler),this;if("object"==typeof a){for(e in a)this.off(e,b,a[e]);return this}return(b===!1||"function"==typeof b)&&(c=b,b=void 0),c===!1&&(c=bb),this.each(function(){m.event.remove(this,a,c,b)})},trigger:function(a,b){return this.each(function(){m.event.trigger(a,b,this)})},triggerHandler:function(a,b){var c=this[0];return c?m.event.trigger(a,b,c,!0):void 0}});function db(a){var b=eb.split("|"),c=a.createDocumentFragment();if(c.createElement)while(b.length)c.createElement(b.pop());return c}var eb="abbr|article|aside|audio|bdi|canvas|data|datalist|details|figcaption|figure|footer|header|hgroup|mark|meter|nav|output|progress|section|summary|time|video",fb=/ jQuery\d+="(?:null|\d+)"/g,gb=new RegExp("<(?:"+eb+")[\\s/>]","i"),hb=/^\s+/,ib=/<(?!area|br|col|embed|hr|img|input|link|meta|param)(([\w:]
+)[^>]*)\/>/gi,jb=/<([\w:]+)/,kb=/<tbody/i,lb=/<|&#?\w+;/,mb=/<(?:script|style|link)/i,nb=/checked\s*(?:[^=]|=\s*.checked.)/i,ob=/^$|\/(?:java|ecma)script/i,pb=/^true\/(.*)/,qb=/^\s*<!(?:\[CDATA\[|--)|(?:\]\]|--)>\s*$/g,rb={option:[1,"<select multiple='multiple'>","</select>"],legend:[1,"<fieldset>","</fieldset>"],area:[1,"<map>","</map>"],param:[1,"<object>","</object>"],thead:[1,"<table>","</table>"],tr:[2,"<table><tbody>","</tbody></table>"],col:[2,"<table><tbody></tbody><colgroup>","</colgroup></table>"],td:[3,"<table><tbody><tr>","</tr></tbody></table>"],_default:k.htmlSerialize?[0,"",""]:[1,"X<div>","</div>"]},sb=db(y),tb=sb.appendChild(y.createElement("div"));rb.optgroup=rb.option,rb.tbody=rb.tfoot=rb.colgroup=rb.caption=rb.thead,rb.th=rb.td;function ub(a,b){var c,d,e=0,f=typeof a.getElementsByTagName!==K?a.getElementsByTagName(b||"*"):typeof a.querySelectorAll!==K?a.querySelectorAll(b||"*"):void 0;if(!f)for(f=[],c=a.childNodes||a;null!=(d=c[e]);e++)!b||m.nodeName(d,b)?f.push
(d):m.merge(f,ub(d,b));return void 0===b||b&&m.nodeName(a,b)?m.merge([a],f):f}function vb(a){W.test(a.type)&&(a.defaultChecked=a.checked)}function wb(a,b){return m.nodeName(a,"table")&&m.nodeName(11!==b.nodeType?b:b.firstChild,"tr")?a.getElementsByTagName("tbody")[0]||a.appendChild(a.ownerDocument.createElement("tbody")):a}function xb(a){return a.type=(null!==m.find.attr(a,"type"))+"/"+a.type,a}function yb(a){var b=pb.exec(a.type);return b?a.type=b[1]:a.removeAttribute("type"),a}function zb(a,b){for(var c,d=0;null!=(c=a[d]);d++)m._data(c,"globalEval",!b||m._data(b[d],"globalEval"))}function Ab(a,b){if(1===b.nodeType&&m.hasData(a)){var c,d,e,f=m._data(a),g=m._data(b,f),h=f.events;if(h){delete g.handle,g.events={};for(c in h)for(d=0,e=h[c].length;e>d;d++)m.event.add(b,c,h[c][d])}g.data&&(g.data=m.extend({},g.data))}}function Bb(a,b){var c,d,e;if(1===b.nodeType){if(c=b.nodeName.toLowerCase(),!k.noCloneEvent&&b[m.expando]){e=m._data(b);for(d in e.events)m.removeEvent(b,d,e.handle);b.rem
oveAttribute(m.expando)}"script"===c&&b.text!==a.text?(xb(b).text=a.text,yb(b)):"object"===c?(b.parentNode&&(b.outerHTML=a.outerHTML),k.html5Clone&&a.innerHTML&&!m.trim(b.innerHTML)&&(b.innerHTML=a.innerHTML)):"input"===c&&W.test(a.type)?(b.defaultChecked=b.checked=a.checked,b.value!==a.value&&(b.value=a.value)):"option"===c?b.defaultSelected=b.selected=a.defaultSelected:("input"===c||"textarea"===c)&&(b.defaultValue=a.defaultValue)}}m.extend({clone:function(a,b,c){var d,e,f,g,h,i=m.contains(a.ownerDocument,a);if(k.html5Clone||m.isXMLDoc(a)||!gb.test("<"+a.nodeName+">")?f=a.cloneNode(!0):(tb.innerHTML=a.outerHTML,tb.removeChild(f=tb.firstChild)),!(k.noCloneEvent&&k.noCloneChecked||1!==a.nodeType&&11!==a.nodeType||m.isXMLDoc(a)))for(d=ub(f),h=ub(a),g=0;null!=(e=h[g]);++g)d[g]&&Bb(e,d[g]);if(b)if(c)for(h=h||ub(a),d=d||ub(f),g=0;null!=(e=h[g]);g++)Ab(e,d[g]);else Ab(a,f);return d=ub(f,"script"),d.length>0&&zb(d,!i&&ub(a,"script")),d=h=e=null,f},buildFragment:function(a,b,c,d){for(var e
,f,g,h,i,j,l,n=a.length,o=db(b),p=[],q=0;n>q;q++)if(f=a[q],f||0===f)if("object"===m.type(f))m.merge(p,f.nodeType?[f]:f);else if(lb.test(f)){h=h||o.appendChild(b.createElement("div")),i=(jb.exec(f)||["",""])[1].toLowerCase(),l=rb[i]||rb._default,h.innerHTML=l[1]+f.replace(ib,"<$1></$2>")+l[2],e=l[0];while(e--)h=h.lastChild;if(!k.leadingWhitespace&&hb.test(f)&&p.push(b.createTextNode(hb.exec(f)[0])),!k.tbody){f="table"!==i||kb.test(f)?"<table>"!==l[1]||kb.test(f)?0:h:h.firstChild,e=f&&f.childNodes.length;while(e--)m.nodeName(j=f.childNodes[e],"tbody")&&!j.childNodes.length&&f.removeChild(j)}m.merge(p,h.childNodes),h.textContent="";while(h.firstChild)h.removeChild(h.firstChild);h=o.lastChild}else p.push(b.createTextNode(f));h&&o.removeChild(h),k.appendChecked||m.grep(ub(p,"input"),vb),q=0;while(f=p[q++])if((!d||-1===m.inArray(f,d))&&(g=m.contains(f.ownerDocument,f),h=ub(o.appendChild(f),"script"),g&&zb(h),c)){e=0;while(f=h[e++])ob.test(f.type||"")&&c.push(f)}return h=null,o},cleanData:
function(a,b){for(var d,e,f,g,h=0,i=m.expando,j=m.cache,l=k.deleteExpando,n=m.event.special;null!=(d=a[h]);h++)if((b||m.acceptData(d))&&(f=d[i],g=f&&j[f])){if(g.events)for(e in g.events)n[e]?m.event.remove(d,e):m.removeEvent(d,e,g.handle);j[f]&&(delete j[f],l?delete d[i]:typeof d.removeAttribute!==K?d.removeAttribute(i):d[i]=null,c.push(f))}}}),m.fn.extend({text:function(a){return V(this,function(a){return void 0===a?m.text(this):this.empty().append((this[0]&&this[0].ownerDocument||y).createTextNode(a))},null,a,arguments.length)},append:function(){return this.domManip(arguments,function(a){if(1===this.nodeType||11===this.nodeType||9===this.nodeType){var b=wb(this,a);b.appendChild(a)}})},prepend:function(){return this.domManip(arguments,function(a){if(1===this.nodeType||11===this.nodeType||9===this.nodeType){var b=wb(this,a);b.insertBefore(a,b.firstChild)}})},before:function(){return this.domManip(arguments,function(a){this.parentNode&&this.parentNode.insertBefore(a,this)})},after:fu
nction(){return this.domManip(arguments,function(a){this.parentNode&&this.parentNode.insertBefore(a,this.nextSibling)})},remove:function(a,b){for(var c,d=a?m.filter(a,this):this,e=0;null!=(c=d[e]);e++)b||1!==c.nodeType||m.cleanData(ub(c)),c.parentNode&&(b&&m.contains(c.ownerDocument,c)&&zb(ub(c,"script")),c.parentNode.removeChild(c));return this},empty:function(){for(var a,b=0;null!=(a=this[b]);b++){1===a.nodeType&&m.cleanData(ub(a,!1));while(a.firstChild)a.removeChild(a.firstChild);a.options&&m.nodeName(a,"select")&&(a.options.length=0)}return this},clone:function(a,b){return a=null==a?!1:a,b=null==b?a:b,this.map(function(){return m.clone(this,a,b)})},html:function(a){return V(this,function(a){var b=this[0]||{},c=0,d=this.length;if(void 0===a)return 1===b.nodeType?b.innerHTML.replace(fb,""):void 0;if(!("string"!=typeof a||mb.test(a)||!k.htmlSerialize&&gb.test(a)||!k.leadingWhitespace&&hb.test(a)||rb[(jb.exec(a)||["",""])[1].toLowerCase()])){a=a.replace(ib,"<$1></$2>");try{for(;d>c;
c++)b=this[c]||{},1===b.nodeType&&(m.cleanData(ub(b,!1)),b.innerHTML=a);b=0}catch(e){}}b&&this.empty().append(a)},null,a,arguments.length)},replaceWith:function(){var a=arguments[0];return this.domManip(arguments,function(b){a=this.parentNode,m.cleanData(ub(this)),a&&a.replaceChild(b,this)}),a&&(a.length||a.nodeType)?this:this.remove()},detach:function(a){return this.remove(a,!0)},domManip:function(a,b){a=e.apply([],a);var c,d,f,g,h,i,j=0,l=this.length,n=this,o=l-1,p=a[0],q=m.isFunction(p);if(q||l>1&&"string"==typeof p&&!k.checkClone&&nb.test(p))return this.each(function(c){var d=n.eq(c);q&&(a[0]=p.call(this,c,d.html())),d.domManip(a,b)});if(l&&(i=m.buildFragment(a,this[0].ownerDocument,!1,this),c=i.firstChild,1===i.childNodes.length&&(i=c),c)){for(g=m.map(ub(i,"script"),xb),f=g.length;l>j;j++)d=i,j!==o&&(d=m.clone(d,!0,!0),f&&m.merge(g,ub(d,"script"))),b.call(this[j],d,j);if(f)for(h=g[g.length-1].ownerDocument,m.map(g,yb),j=0;f>j;j++)d=g[j],ob.test(d.type||"")&&!m._data(d,"globalEv
al")&&m.contains(h,d)&&(d.src?m._evalUrl&&m._evalUrl(d.src):m.globalEval((d.text||d.textContent||d.innerHTML||"").replace(qb,"")));i=c=null}return this}}),m.each({appendTo:"append",prependTo:"prepend",insertBefore:"before",insertAfter:"after",replaceAll:"replaceWith"},function(a,b){m.fn[a]=function(a){for(var c,d=0,e=[],g=m(a),h=g.length-1;h>=d;d++)c=d===h?this:this.clone(!0),m(g[d])[b](c),f.apply(e,c.get());return this.pushStack(e)}});var Cb,Db={};function Eb(b,c){var d,e=m(c.createElement(b)).appendTo(c.body),f=a.getDefaultComputedStyle&&(d=a.getDefaultComputedStyle(e[0]))?d.display:m.css(e[0],"display");return e.detach(),f}function Fb(a){var b=y,c=Db[a];return c||(c=Eb(a,b),"none"!==c&&c||(Cb=(Cb||m("<iframe frameborder='0' width='0' height='0'/>")).appendTo(b.documentElement),b=(Cb[0].contentWindow||Cb[0].contentDocument).document,b.write(),b.close(),c=Eb(a,b),Cb.detach()),Db[a]=c),c}!function(){var a;k.shrinkWrapBlocks=function(){if(null!=a)return a;a=!1;var b,c,d;return c=y.ge
tElementsByTagName("body")[0],c&&c.style?(b=y.createElement("div"),d=y.createElement("div"),d.style.cssText="position:absolute;border:0;width:0;height:0;top:0;left:-9999px",c.appendChild(d).appendChild(b),typeof b.style.zoom!==K&&(b.style.cssText="-webkit-box-sizing:content-box;-moz-box-sizing:content-box;box-sizing:content-box;display:block;margin:0;border:0;padding:1px;width:1px;zoom:1",b.appendChild(y.createElement("div")).style.width="5px",a=3!==b.offsetWidth),c.removeChild(d),a):void 0}}();var Gb=/^margin/,Hb=new RegExp("^("+S+")(?!px)[a-z%]+$","i"),Ib,Jb,Kb=/^(top|right|bottom|left)$/;a.getComputedStyle?(Ib=function(a){return a.ownerDocument.defaultView.getComputedStyle(a,null)},Jb=function(a,b,c){var d,e,f,g,h=a.style;return c=c||Ib(a),g=c?c.getPropertyValue(b)||c[b]:void 0,c&&(""!==g||m.contains(a.ownerDocument,a)||(g=m.style(a,b)),Hb.test(g)&&Gb.test(b)&&(d=h.width,e=h.minWidth,f=h.maxWidth,h.minWidth=h.maxWidth=h.width=g,g=c.width,h.width=d,h.minWidth=e,h.maxWidth=f)),void
0===g?g:g+""}):y.documentElement.currentStyle&&(Ib=function(a){return a.currentStyle},Jb=function(a,b,c){var d,e,f,g,h=a.style;return c=c||Ib(a),g=c?c[b]:void 0,null==g&&h&&h[b]&&(g=h[b]),Hb.test(g)&&!Kb.test(b)&&(d=h.left,e=a.runtimeStyle,f=e&&e.left,f&&(e.left=a.currentStyle.left),h.left="fontSize"===b?"1em":g,g=h.pixelLeft+"px",h.left=d,f&&(e.left=f)),void 0===g?g:g+""||"auto"});function Lb(a,b){return{get:function(){var c=a();if(null!=c)return c?void delete this.get:(this.get=b).apply(this,arguments)}}}!function(){var b,c,d,e,f,g,h;if(b=y.createElement("div"),b.innerHTML=" <link/><table></table><a href='/a'>a</a><input type='checkbox'/>",d=b.getElementsByTagName("a")[0],c=d&&d.style){c.cssText="float:left;opacity:.5",k.opacity="0.5"===c.opacity,k.cssFloat=!!c.cssFloat,b.style.backgroundClip="content-box",b.cloneNode(!0).style.backgroundClip="",k.clearCloneStyle="content-box"===b.style.backgroundClip,k.boxSizing=""===c.boxSizing||""===c.MozBoxSizing||""===c.WebkitBoxSizing,m.ex
tend(k,{reliableHiddenOffsets:function(){return null==g&&i(),g},boxSizingReliable:function(){return null==f&&i(),f},pixelPosition:function(){return null==e&&i(),e},reliableMarginRight:function(){return null==h&&i(),h}});function i(){var b,c,d,i;c=y.getElementsByTagName("body")[0],c&&c.style&&(b=y.createElement("div"),d=y.createElement("div"),d.style.cssText="position:absolute;border:0;width:0;height:0;top:0;left:-9999px",c.appendChild(d).appendChild(b),b.style.cssText="-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box;display:block;margin-top:1%;top:1%;border:1px;padding:1px;width:4px;position:absolute",e=f=!1,h=!0,a.getComputedStyle&&(e="1%"!==(a.getComputedStyle(b,null)||{}).top,f="4px"===(a.getComputedStyle(b,null)||{width:"4px"}).width,i=b.appendChild(y.createElement("div")),i.style.cssText=b.style.cssText="-webkit-box-sizing:content-box;-moz-box-sizing:content-box;box-sizing:content-box;display:block;margin:0;border:0;padding:0",i.style.marginRight=
i.style.width="0",b.style.width="1px",h=!parseFloat((a.getComputedStyle(i,null)||{}).marginRight)),b.innerHTML="<table><tr><td></td><td>t</td></tr></table>",i=b.getElementsByTagName("td"),i[0].style.cssText="margin:0;border:0;padding:0;display:none",g=0===i[0].offsetHeight,g&&(i[0].style.display="",i[1].style.display="none",g=0===i[0].offsetHeight),c.removeChild(d))}}}(),m.swap=function(a,b,c,d){var e,f,g={};for(f in b)g[f]=a.style[f],a.style[f]=b[f];e=c.apply(a,d||[]);for(f in b)a.style[f]=g[f];return e};var Mb=/alpha\([^)]*\)/i,Nb=/opacity\s*=\s*([^)]*)/,Ob=/^(none|table(?!-c[ea]).+)/,Pb=new RegExp("^("+S+")(.*)$","i"),Qb=new RegExp("^([+-])=("+S+")","i"),Rb={position:"absolute",visibility:"hidden",display:"block"},Sb={letterSpacing:"0",fontWeight:"400"},Tb=["Webkit","O","Moz","ms"];function Ub(a,b){if(b in a)return b;var c=b.charAt(0).toUpperCase()+b.slice(1),d=b,e=Tb.length;while(e--)if(b=Tb[e]+c,b in a)return b;return d}function Vb(a,b){for(var c,d,e,f=[],g=0,h=a.length;h>g;g++
)d=a[g],d.style&&(f[g]=m._data(d,"olddisplay"),c=d.style.display,b?(f[g]||"none"!==c||(d.style.display=""),""===d.style.display&&U(d)&&(f[g]=m._data(d,"olddisplay",Fb(d.nodeName)))):(e=U(d),(c&&"none"!==c||!e)&&m._data(d,"olddisplay",e?c:m.css(d,"display"))));for(g=0;h>g;g++)d=a[g],d.style&&(b&&"none"!==d.style.display&&""!==d.style.display||(d.style.display=b?f[g]||"":"none"));return a}function Wb(a,b,c){var d=Pb.exec(b);return d?Math.max(0,d[1]-(c||0))+(d[2]||"px"):b}function Xb(a,b,c,d,e){for(var f=c===(d?"border":"content")?4:"width"===b?1:0,g=0;4>f;f+=2)"margin"===c&&(g+=m.css(a,c+T[f],!0,e)),d?("content"===c&&(g-=m.css(a,"padding"+T[f],!0,e)),"margin"!==c&&(g-=m.css(a,"border"+T[f]+"Width",!0,e))):(g+=m.css(a,"padding"+T[f],!0,e),"padding"!==c&&(g+=m.css(a,"border"+T[f]+"Width",!0,e)));return g}function Yb(a,b,c){var d=!0,e="width"===b?a.offsetWidth:a.offsetHeight,f=Ib(a),g=k.boxSizing&&"border-box"===m.css(a,"boxSizing",!1,f);if(0>=e||null==e){if(e=Jb(a,b,f),(0>e||null==e)&&(
e=a.style[b]),Hb.test(e))return e;d=g&&(k.boxSizingReliable()||e===a.style[b]),e=parseFloat(e)||0}return e+Xb(a,b,c||(g?"border":"content"),d,f)+"px"}m.extend({cssHooks:{opacity:{get:function(a,b){if(b){var c=Jb(a,"opacity");return""===c?"1":c}}}},cssNumber:{columnCount:!0,fillOpacity:!0,flexGrow:!0,flexShrink:!0,fontWeight:!0,lineHeight:!0,opacity:!0,order:!0,orphans:!0,widows:!0,zIndex:!0,zoom:!0},cssProps:{"float":k.cssFloat?"cssFloat":"styleFloat"},style:function(a,b,c,d){if(a&&3!==a.nodeType&&8!==a.nodeType&&a.style){var e,f,g,h=m.camelCase(b),i=a.style;if(b=m.cssProps[h]||(m.cssProps[h]=Ub(i,h)),g=m.cssHooks[b]||m.cssHooks[h],void 0===c)return g&&"get"in g&&void 0!==(e=g.get(a,!1,d))?e:i[b];if(f=typeof c,"string"===f&&(e=Qb.exec(c))&&(c=(e[1]+1)*e[2]+parseFloat(m.css(a,b)),f="number"),null!=c&&c===c&&("number"!==f||m.cssNumber[h]||(c+="px"),k.clearCloneStyle||""!==c||0!==b.indexOf("background")||(i[b]="inherit"),!(g&&"set"in g&&void 0===(c=g.set(a,c,d)))))try{i[b]=c}catch(j){}
}},css:function(a,b,c,d){var e,f,g,h=m.camelCase(b);return b=m.cssProps[h]||(m.cssProps[h]=Ub(a.style,h)),g=m.cssHooks[b]||m.cssHooks[h],g&&"get"in g&&(f=g.get(a,!0,c)),void 0===f&&(f=Jb(a,b,d)),"normal"===f&&b in Sb&&(f=Sb[b]),""===c||c?(e=parseFloat(f),c===!0||m.isNumeric(e)?e||0:f):f}}),m.each(["height","width"],function(a,b){m.cssHooks[b]={get:function(a,c,d){return c?Ob.test(m.css(a,"display"))&&0===a.offsetWidth?m.swap(a,Rb,function(){return Yb(a,b,d)}):Yb(a,b,d):void 0},set:function(a,c,d){var e=d&&Ib(a);return Wb(a,c,d?Xb(a,b,d,k.boxSizing&&"border-box"===m.css(a,"boxSizing",!1,e),e):0)}}}),k.opacity||(m.cssHooks.opacity={get:function(a,b){return Nb.test((b&&a.currentStyle?a.currentStyle.filter:a.style.filter)||"")?.01*parseFloat(RegExp.$1)+"":b?"1":""},set:function(a,b){var c=a.style,d=a.currentStyle,e=m.isNumeric(b)?"alpha(opacity="+100*b+")":"",f=d&&d.filter||c.filter||"";c.zoom=1,(b>=1||""===b)&&""===m.trim(f.replace(Mb,""))&&c.removeAttribute&&(c.removeAttribute("filter
"),""===b||d&&!d.filter)||(c.filter=Mb.test(f)?f.replace(Mb,e):f+" "+e)}}),m.cssHooks.marginRight=Lb(k.reliableMarginRight,function(a,b){return b?m.swap(a,{display:"inline-block"},Jb,[a,"marginRight"]):void 0}),m.each({margin:"",padding:"",border:"Width"},function(a,b){m.cssHooks[a+b]={expand:function(c){for(var d=0,e={},f="string"==typeof c?c.split(" "):[c];4>d;d++)e[a+T[d]+b]=f[d]||f[d-2]||f[0];return e}},Gb.test(a)||(m.cssHooks[a+b].set=Wb)}),m.fn.extend({css:function(a,b){return V(this,function(a,b,c){var d,e,f={},g=0;if(m.isArray(b)){for(d=Ib(a),e=b.length;e>g;g++)f[b[g]]=m.css(a,b[g],!1,d);return f}return void 0!==c?m.style(a,b,c):m.css(a,b)},a,b,arguments.length>1)},show:function(){return Vb(this,!0)},hide:function(){return Vb(this)},toggle:function(a){return"boolean"==typeof a?a?this.show():this.hide():this.each(function(){U(this)?m(this).show():m(this).hide()})}});function Zb(a,b,c,d,e){return new Zb.prototype.init(a,b,c,d,e)}m.Tween=Zb,Zb.prototype={constructor:Zb,init:fun
ction(a,b,c,d,e,f){this.elem=a,this.prop=c,this.easing=e||"swing",this.options=b,this.start=this.now=this.cur(),this.end=d,this.unit=f||(m.cssNumber[c]?"":"px")
+},cur:function(){var a=Zb.propHooks[this.prop];return a&&a.get?a.get(this):Zb.propHooks._default.get(this)},run:function(a){var b,c=Zb.propHooks[this.prop];return this.pos=b=this.options.duration?m.easing[this.easing](a,this.options.duration*a,0,1,this.options.duration):a,this.now=(this.end-this.start)*b+this.start,this.options.step&&this.options.step.call(this.elem,this.now,this),c&&c.set?c.set(this):Zb.propHooks._default.set(this),this}},Zb.prototype.init.prototype=Zb.prototype,Zb.propHooks={_default:{get:function(a){var b;return null==a.elem[a.prop]||a.elem.style&&null!=a.elem.style[a.prop]?(b=m.css(a.elem,a.prop,""),b&&"auto"!==b?b:0):a.elem[a.prop]},set:function(a){m.fx.step[a.prop]?m.fx.step[a.prop](a):a.elem.style&&(null!=a.elem.style[m.cssProps[a.prop]]||m.cssHooks[a.prop])?m.style(a.elem,a.prop,a.now+a.unit):a.elem[a.prop]=a.now}}},Zb.propHooks.scrollTop=Zb.propHooks.scrollLeft={set:function(a){a.elem.nodeType&&a.elem.parentNode&&(a.elem[a.prop]=a.now)}},m.easing={linear:fu
nction(a){return a},swing:function(a){return.5-Math.cos(a*Math.PI)/2}},m.fx=Zb.prototype.init,m.fx.step={};var $b,_b,ac=/^(?:toggle|show|hide)$/,bc=new RegExp("^(?:([+-])=|)("+S+")([a-z%]*)$","i"),cc=/queueHooks$/,dc=[ic],ec={"*":[function(a,b){var c=this.createTween(a,b),d=c.cur(),e=bc.exec(b),f=e&&e[3]||(m.cssNumber[a]?"":"px"),g=(m.cssNumber[a]||"px"!==f&&+d)&&bc.exec(m.css(c.elem,a)),h=1,i=20;if(g&&g[3]!==f){f=f||g[3],e=e||[],g=+d||1;do h=h||".5",g/=h,m.style(c.elem,a,g+f);while(h!==(h=c.cur()/d)&&1!==h&&--i)}return e&&(g=c.start=+g||+d||0,c.unit=f,c.end=e[1]?g+(e[1]+1)*e[2]:+e[2]),c}]};function fc(){return setTimeout(function(){$b=void 0}),$b=m.now()}function gc(a,b){var c,d={height:a},e=0;for(b=b?1:0;4>e;e+=2-b)c=T[e],d["margin"+c]=d["padding"+c]=a;return b&&(d.opacity=d.width=a),d}function hc(a,b,c){for(var d,e=(ec[b]||[]).concat(ec["*"]),f=0,g=e.length;g>f;f++)if(d=e[f].call(c,b,a))return d}function ic(a,b,c){var d,e,f,g,h,i,j,l,n=this,o={},p=a.style,q=a.nodeType&&U(a),r=m._
data(a,"fxshow");c.queue||(h=m._queueHooks(a,"fx"),null==h.unqueued&&(h.unqueued=0,i=h.empty.fire,h.empty.fire=function(){h.unqueued||i()}),h.unqueued++,n.always(function(){n.always(function(){h.unqueued--,m.queue(a,"fx").length||h.empty.fire()})})),1===a.nodeType&&("height"in b||"width"in b)&&(c.overflow=[p.overflow,p.overflowX,p.overflowY],j=m.css(a,"display"),l="none"===j?m._data(a,"olddisplay")||Fb(a.nodeName):j,"inline"===l&&"none"===m.css(a,"float")&&(k.inlineBlockNeedsLayout&&"inline"!==Fb(a.nodeName)?p.zoom=1:p.display="inline-block")),c.overflow&&(p.overflow="hidden",k.shrinkWrapBlocks()||n.always(function(){p.overflow=c.overflow[0],p.overflowX=c.overflow[1],p.overflowY=c.overflow[2]}));for(d in b)if(e=b[d],ac.exec(e)){if(delete b[d],f=f||"toggle"===e,e===(q?"hide":"show")){if("show"!==e||!r||void 0===r[d])continue;q=!0}o[d]=r&&r[d]||m.style(a,d)}else j=void 0;if(m.isEmptyObject(o))"inline"===("none"===j?Fb(a.nodeName):j)&&(p.display=j);else{r?"hidden"in r&&(q=r.hidden):r=m
._data(a,"fxshow",{}),f&&(r.hidden=!q),q?m(a).show():n.done(function(){m(a).hide()}),n.done(function(){var b;m._removeData(a,"fxshow");for(b in o)m.style(a,b,o[b])});for(d in o)g=hc(q?r[d]:0,d,n),d in r||(r[d]=g.start,q&&(g.end=g.start,g.start="width"===d||"height"===d?1:0))}}function jc(a,b){var c,d,e,f,g;for(c in a)if(d=m.camelCase(c),e=b[d],f=a[c],m.isArray(f)&&(e=f[1],f=a[c]=f[0]),c!==d&&(a[d]=f,delete a[c]),g=m.cssHooks[d],g&&"expand"in g){f=g.expand(f),delete a[d];for(c in f)c in a||(a[c]=f[c],b[c]=e)}else b[d]=e}function kc(a,b,c){var d,e,f=0,g=dc.length,h=m.Deferred().always(function(){delete i.elem}),i=function(){if(e)return!1;for(var b=$b||fc(),c=Math.max(0,j.startTime+j.duration-b),d=c/j.duration||0,f=1-d,g=0,i=j.tweens.length;i>g;g++)j.tweens[g].run(f);return h.notifyWith(a,[j,f,c]),1>f&&i?c:(h.resolveWith(a,[j]),!1)},j=h.promise({elem:a,props:m.extend({},b),opts:m.extend(!0,{specialEasing:{}},c),originalProperties:b,originalOptions:c,startTime:$b||fc(),duration:c.durati
on,tweens:[],createTween:function(b,c){var d=m.Tween(a,j.opts,b,c,j.opts.specialEasing[b]||j.opts.easing);return j.tweens.push(d),d},stop:function(b){var c=0,d=b?j.tweens.length:0;if(e)return this;for(e=!0;d>c;c++)j.tweens[c].run(1);return b?h.resolveWith(a,[j,b]):h.rejectWith(a,[j,b]),this}}),k=j.props;for(jc(k,j.opts.specialEasing);g>f;f++)if(d=dc[f].call(j,a,k,j.opts))return d;return m.map(k,hc,j),m.isFunction(j.opts.start)&&j.opts.start.call(a,j),m.fx.timer(m.extend(i,{elem:a,anim:j,queue:j.opts.queue})),j.progress(j.opts.progress).done(j.opts.done,j.opts.complete).fail(j.opts.fail).always(j.opts.always)}m.Animation=m.extend(kc,{tweener:function(a,b){m.isFunction(a)?(b=a,a=["*"]):a=a.split(" ");for(var c,d=0,e=a.length;e>d;d++)c=a[d],ec[c]=ec[c]||[],ec[c].unshift(b)},prefilter:function(a,b){b?dc.unshift(a):dc.push(a)}}),m.speed=function(a,b,c){var d=a&&"object"==typeof a?m.extend({},a):{complete:c||!c&&b||m.isFunction(a)&&a,duration:a,easing:c&&b||b&&!m.isFunction(b)&&b};return
d.duration=m.fx.off?0:"number"==typeof d.duration?d.duration:d.duration in m.fx.speeds?m.fx.speeds[d.duration]:m.fx.speeds._default,(null==d.queue||d.queue===!0)&&(d.queue="fx"),d.old=d.complete,d.complete=function(){m.isFunction(d.old)&&d.old.call(this),d.queue&&m.dequeue(this,d.queue)},d},m.fn.extend({fadeTo:function(a,b,c,d){return this.filter(U).css("opacity",0).show().end().animate({opacity:b},a,c,d)},animate:function(a,b,c,d){var e=m.isEmptyObject(a),f=m.speed(b,c,d),g=function(){var b=kc(this,m.extend({},a),f);(e||m._data(this,"finish"))&&b.stop(!0)};return g.finish=g,e||f.queue===!1?this.each(g):this.queue(f.queue,g)},stop:function(a,b,c){var d=function(a){var b=a.stop;delete a.stop,b(c)};return"string"!=typeof a&&(c=b,b=a,a=void 0),b&&a!==!1&&this.queue(a||"fx",[]),this.each(function(){var b=!0,e=null!=a&&a+"queueHooks",f=m.timers,g=m._data(this);if(e)g[e]&&g[e].stop&&d(g[e]);else for(e in g)g[e]&&g[e].stop&&cc.test(e)&&d(g[e]);for(e=f.length;e--;)f[e].elem!==this||null!=a&
&f[e].queue!==a||(f[e].anim.stop(c),b=!1,f.splice(e,1));(b||!c)&&m.dequeue(this,a)})},finish:function(a){return a!==!1&&(a=a||"fx"),this.each(function(){var b,c=m._data(this),d=c[a+"queue"],e=c[a+"queueHooks"],f=m.timers,g=d?d.length:0;for(c.finish=!0,m.queue(this,a,[]),e&&e.stop&&e.stop.call(this,!0),b=f.length;b--;)f[b].elem===this&&f[b].queue===a&&(f[b].anim.stop(!0),f.splice(b,1));for(b=0;g>b;b++)d[b]&&d[b].finish&&d[b].finish.call(this);delete c.finish})}}),m.each(["toggle","show","hide"],function(a,b){var c=m.fn[b];m.fn[b]=function(a,d,e){return null==a||"boolean"==typeof a?c.apply(this,arguments):this.animate(gc(b,!0),a,d,e)}}),m.each({slideDown:gc("show"),slideUp:gc("hide"),slideToggle:gc("toggle"),fadeIn:{opacity:"show"},fadeOut:{opacity:"hide"},fadeToggle:{opacity:"toggle"}},function(a,b){m.fn[a]=function(a,c,d){return this.animate(b,a,c,d)}}),m.timers=[],m.fx.tick=function(){var a,b=m.timers,c=0;for($b=m.now();c<b.length;c++)a=b[c],a()||b[c]!==a||b.splice(c--,1);b.length|
|m.fx.stop(),$b=void 0},m.fx.timer=function(a){m.timers.push(a),a()?m.fx.start():m.timers.pop()},m.fx.interval=13,m.fx.start=function(){_b||(_b=setInterval(m.fx.tick,m.fx.interval))},m.fx.stop=function(){clearInterval(_b),_b=null},m.fx.speeds={slow:600,fast:200,_default:400},m.fn.delay=function(a,b){return a=m.fx?m.fx.speeds[a]||a:a,b=b||"fx",this.queue(b,function(b,c){var d=setTimeout(b,a);c.stop=function(){clearTimeout(d)}})},function(){var a,b,c,d,e;b=y.createElement("div"),b.setAttribute("className","t"),b.innerHTML=" <link/><table></table><a href='/a'>a</a><input type='checkbox'/>",d=b.getElementsByTagName("a")[0],c=y.createElement("select"),e=c.appendChild(y.createElement("option")),a=b.getElementsByTagName("input")[0],d.style.cssText="top:1px",k.getSetAttribute="t"!==b.className,k.style=/top/.test(d.getAttribute("style")),k.hrefNormalized="/a"===d.getAttribute("href"),k.checkOn=!!a.value,k.optSelected=e.selected,k.enctype=!!y.createElement("form").enctype,c.disabled=!0,k.opt
Disabled=!e.disabled,a=y.createElement("input"),a.setAttribute("value",""),k.input=""===a.getAttribute("value"),a.value="t",a.setAttribute("type","radio"),k.radioValue="t"===a.value}();var lc=/\r/g;m.fn.extend({val:function(a){var b,c,d,e=this[0];{if(arguments.length)return d=m.isFunction(a),this.each(function(c){var e;1===this.nodeType&&(e=d?a.call(this,c,m(this).val()):a,null==e?e="":"number"==typeof e?e+="":m.isArray(e)&&(e=m.map(e,function(a){return null==a?"":a+""})),b=m.valHooks[this.type]||m.valHooks[this.nodeName.toLowerCase()],b&&"set"in b&&void 0!==b.set(this,e,"value")||(this.value=e))});if(e)return b=m.valHooks[e.type]||m.valHooks[e.nodeName.toLowerCase()],b&&"get"in b&&void 0!==(c=b.get(e,"value"))?c:(c=e.value,"string"==typeof c?c.replace(lc,""):null==c?"":c)}}}),m.extend({valHooks:{option:{get:function(a){var b=m.find.attr(a,"value");return null!=b?b:m.trim(m.text(a))}},select:{get:function(a){for(var b,c,d=a.options,e=a.selectedIndex,f="select-one"===a.type||0>e,g=f?
null:[],h=f?e+1:d.length,i=0>e?h:f?e:0;h>i;i++)if(c=d[i],!(!c.selected&&i!==e||(k.optDisabled?c.disabled:null!==c.getAttribute("disabled"))||c.parentNode.disabled&&m.nodeName(c.parentNode,"optgroup"))){if(b=m(c).val(),f)return b;g.push(b)}return g},set:function(a,b){var c,d,e=a.options,f=m.makeArray(b),g=e.length;while(g--)if(d=e[g],m.inArray(m.valHooks.option.get(d),f)>=0)try{d.selected=c=!0}catch(h){d.scrollHeight}else d.selected=!1;return c||(a.selectedIndex=-1),e}}}}),m.each(["radio","checkbox"],function(){m.valHooks[this]={set:function(a,b){return m.isArray(b)?a.checked=m.inArray(m(a).val(),b)>=0:void 0}},k.checkOn||(m.valHooks[this].get=function(a){return null===a.getAttribute("value")?"on":a.value})});var mc,nc,oc=m.expr.attrHandle,pc=/^(?:checked|selected)$/i,qc=k.getSetAttribute,rc=k.input;m.fn.extend({attr:function(a,b){return V(this,m.attr,a,b,arguments.length>1)},removeAttr:function(a){return this.each(function(){m.removeAttr(this,a)})}}),m.extend({attr:function(a,b,c){v
ar d,e,f=a.nodeType;if(a&&3!==f&&8!==f&&2!==f)return typeof a.getAttribute===K?m.prop(a,b,c):(1===f&&m.isXMLDoc(a)||(b=b.toLowerCase(),d=m.attrHooks[b]||(m.expr.match.bool.test(b)?nc:mc)),void 0===c?d&&"get"in d&&null!==(e=d.get(a,b))?e:(e=m.find.attr(a,b),null==e?void 0:e):null!==c?d&&"set"in d&&void 0!==(e=d.set(a,c,b))?e:(a.setAttribute(b,c+""),c):void m.removeAttr(a,b))},removeAttr:function(a,b){var c,d,e=0,f=b&&b.match(E);if(f&&1===a.nodeType)while(c=f[e++])d=m.propFix[c]||c,m.expr.match.bool.test(c)?rc&&qc||!pc.test(c)?a[d]=!1:a[m.camelCase("default-"+c)]=a[d]=!1:m.attr(a,c,""),a.removeAttribute(qc?c:d)},attrHooks:{type:{set:function(a,b){if(!k.radioValue&&"radio"===b&&m.nodeName(a,"input")){var c=a.value;return a.setAttribute("type",b),c&&(a.value=c),b}}}}}),nc={set:function(a,b,c){return b===!1?m.removeAttr(a,c):rc&&qc||!pc.test(c)?a.setAttribute(!qc&&m.propFix[c]||c,c):a[m.camelCase("default-"+c)]=a[c]=!0,c}},m.each(m.expr.match.bool.source.match(/\w+/g),function(a,b){var c
=oc[b]||m.find.attr;oc[b]=rc&&qc||!pc.test(b)?function(a,b,d){var e,f;return d||(f=oc[b],oc[b]=e,e=null!=c(a,b,d)?b.toLowerCase():null,oc[b]=f),e}:function(a,b,c){return c?void 0:a[m.camelCase("default-"+b)]?b.toLowerCase():null}}),rc&&qc||(m.attrHooks.value={set:function(a,b,c){return m.nodeName(a,"input")?void(a.defaultValue=b):mc&&mc.set(a,b,c)}}),qc||(mc={set:function(a,b,c){var d=a.getAttributeNode(c);return d||a.setAttributeNode(d=a.ownerDocument.createAttribute(c)),d.value=b+="","value"===c||b===a.getAttribute(c)?b:void 0}},oc.id=oc.name=oc.coords=function(a,b,c){var d;return c?void 0:(d=a.getAttributeNode(b))&&""!==d.value?d.value:null},m.valHooks.button={get:function(a,b){var c=a.getAttributeNode(b);return c&&c.specified?c.value:void 0},set:mc.set},m.attrHooks.contenteditable={set:function(a,b,c){mc.set(a,""===b?!1:b,c)}},m.each(["width","height"],function(a,b){m.attrHooks[b]={set:function(a,c){return""===c?(a.setAttribute(b,"auto"),c):void 0}}})),k.style||(m.attrHooks.styl
e={get:function(a){return a.style.cssText||void 0},set:function(a,b){return a.style.cssText=b+""}});var sc=/^(?:input|select|textarea|button|object)$/i,tc=/^(?:a|area)$/i;m.fn.extend({prop:function(a,b){return V(this,m.prop,a,b,arguments.length>1)},removeProp:function(a){return a=m.propFix[a]||a,this.each(function(){try{this[a]=void 0,delete this[a]}catch(b){}})}}),m.extend({propFix:{"for":"htmlFor","class":"className"},prop:function(a,b,c){var d,e,f,g=a.nodeType;if(a&&3!==g&&8!==g&&2!==g)return f=1!==g||!m.isXMLDoc(a),f&&(b=m.propFix[b]||b,e=m.propHooks[b]),void 0!==c?e&&"set"in e&&void 0!==(d=e.set(a,c,b))?d:a[b]=c:e&&"get"in e&&null!==(d=e.get(a,b))?d:a[b]},propHooks:{tabIndex:{get:function(a){var b=m.find.attr(a,"tabindex");return b?parseInt(b,10):sc.test(a.nodeName)||tc.test(a.nodeName)&&a.href?0:-1}}}}),k.hrefNormalized||m.each(["href","src"],function(a,b){m.propHooks[b]={get:function(a){return a.getAttribute(b,4)}}}),k.optSelected||(m.propHooks.selected={get:function(a){var b
=a.parentNode;return b&&(b.selectedIndex,b.parentNode&&b.parentNode.selectedIndex),null}}),m.each(["tabIndex","readOnly","maxLength","cellSpacing","cellPadding","rowSpan","colSpan","useMap","frameBorder","contentEditable"],function(){m.propFix[this.toLowerCase()]=this}),k.enctype||(m.propFix.enctype="encoding");var uc=/[\t\r\n\f]/g;m.fn.extend({addClass:function(a){var b,c,d,e,f,g,h=0,i=this.length,j="string"==typeof a&&a;if(m.isFunction(a))return this.each(function(b){m(this).addClass(a.call(this,b,this.className))});if(j)for(b=(a||"").match(E)||[];i>h;h++)if(c=this[h],d=1===c.nodeType&&(c.className?(" "+c.className+" ").replace(uc," "):" ")){f=0;while(e=b[f++])d.indexOf(" "+e+" ")<0&&(d+=e+" ");g=m.trim(d),c.className!==g&&(c.className=g)}return this},removeClass:function(a){var b,c,d,e,f,g,h=0,i=this.length,j=0===arguments.length||"string"==typeof a&&a;if(m.isFunction(a))return this.each(function(b){m(this).removeClass(a.call(this,b,this.className))});if(j)for(b=(a||"").match(E)|
|[];i>h;h++)if(c=this[h],d=1===c.nodeType&&(c.className?(" "+c.className+" ").replace(uc," "):"")){f=0;while(e=b[f++])while(d.indexOf(" "+e+" ")>=0)d=d.replace(" "+e+" "," ");g=a?m.trim(d):"",c.className!==g&&(c.className=g)}return this},toggleClass:function(a,b){var c=typeof a;return"boolean"==typeof b&&"string"===c?b?this.addClass(a):this.removeClass(a):this.each(m.isFunction(a)?function(c){m(this).toggleClass(a.call(this,c,this.className,b),b)}:function(){if("string"===c){var b,d=0,e=m(this),f=a.match(E)||[];while(b=f[d++])e.hasClass(b)?e.removeClass(b):e.addClass(b)}else(c===K||"boolean"===c)&&(this.className&&m._data(this,"__className__",this.className),this.className=this.className||a===!1?"":m._data(this,"__className__")||"")})},hasClass:function(a){for(var b=" "+a+" ",c=0,d=this.length;d>c;c++)if(1===this[c].nodeType&&(" "+this[c].className+" ").replace(uc," ").indexOf(b)>=0)return!0;return!1}}),m.each("blur focus focusin focusout load resize scroll unload click dblclick mou
sedown mouseup mousemove mouseover mouseout mouseenter mouseleave change select submit keydown keypress keyup error contextmenu".split(" "),function(a,b){m.fn[b]=function(a,c){return arguments.length>0?this.on(b,null,a,c):this.trigger(b)}}),m.fn.extend({hover:function(a,b){return this.mouseenter(a).mouseleave(b||a)},bind:function(a,b,c){return this.on(a,null,b,c)},unbind:function(a,b){return this.off(a,null,b)},delegate:function(a,b,c,d){return this.on(b,a,c,d)},undelegate:function(a,b,c){return 1===arguments.length?this.off(a,"**"):this.off(b,a||"**",c)}});var vc=m.now(),wc=/\?/,xc=/(,)|(\[|{)|(}|])|"(?:[^"\\\r\n]|\\["\\\/bfnrt]|\\u[\da-fA-F]{4})*"\s*:?|true|false|null|-?(?!0\d)\d+(?:\.\d+|)(?:[eE][+-]?\d+|)/g;m.parseJSON=function(b){if(a.JSON&&a.JSON.parse)return a.JSON.parse(b+"");var c,d=null,e=m.trim(b+"");return e&&!m.trim(e.replace(xc,function(a,b,e,f){return c&&b&&(d=0),0===d?a:(c=e||b,d+=!f-!e,"")}))?Function("return "+e)():m.error("Invalid JSON: "+b)},m.parseXML=function(b
){var c,d;if(!b||"string"!=typeof b)return null;try{a.DOMParser?(d=new DOMParser,c=d.parseFromString(b,"text/xml")):(c=new ActiveXObject("Microsoft.XMLDOM"),c.async="false",c.loadXML(b))}catch(e){c=void 0}return c&&c.documentElement&&!c.getElementsByTagName("parsererror").length||m.error("Invalid XML: "+b),c};var yc,zc,Ac=/#.*$/,Bc=/([?&])_=[^&]*/,Cc=/^(.*?):[ \t]*([^\r\n]*)\r?$/gm,Dc=/^(?:about|app|app-storage|.+-extension|file|res|widget):$/,Ec=/^(?:GET|HEAD)$/,Fc=/^\/\//,Gc=/^([\w.+-]+:)(?:\/\/(?:[^\/?#]*@|)([^\/?#:]*)(?::(\d+)|)|)/,Hc={},Ic={},Jc="*/".concat("*");try{zc=location.href}catch(Kc){zc=y.createElement("a"),zc.href="",zc=zc.href}yc=Gc.exec(zc.toLowerCase())||[];function Lc(a){return function(b,c){"string"!=typeof b&&(c=b,b="*");var d,e=0,f=b.toLowerCase().match(E)||[];if(m.isFunction(c))while(d=f[e++])"+"===d.charAt(0)?(d=d.slice(1)||"*",(a[d]=a[d]||[]).unshift(c)):(a[d]=a[d]||[]).push(c)}}function Mc(a,b,c,d){var e={},f=a===Ic;function g(h){var i;return e[h]=!0,m.each
(a[h]||[],function(a,h){var j=h(b,c,d);return"string"!=typeof j||f||e[j]?f?!(i=j):void 0:(b.dataTypes.unshift(j),g(j),!1)}),i}return g(b.dataTypes[0])||!e["*"]&&g("*")}function Nc(a,b){var c,d,e=m.ajaxSettings.flatOptions||{};for(d in b)void 0!==b[d]&&((e[d]?a:c||(c={}))[d]=b[d]);return c&&m.extend(!0,a,c),a}function Oc(a,b,c){var d,e,f,g,h=a.contents,i=a.dataTypes;while("*"===i[0])i.shift(),void 0===e&&(e=a.mimeType||b.getResponseHeader("Content-Type"));if(e)for(g in h)if(h[g]&&h[g].test(e)){i.unshift(g);break}if(i[0]in c)f=i[0];else{for(g in c){if(!i[0]||a.converters[g+" "+i[0]]){f=g;break}d||(d=g)}f=f||d}return f?(f!==i[0]&&i.unshift(f),c[f]):void 0}function Pc(a,b,c,d){var e,f,g,h,i,j={},k=a.dataTypes.slice();if(k[1])for(g in a.converters)j[g.toLowerCase()]=a.converters[g];f=k.shift();while(f)if(a.responseFields[f]&&(c[a.responseFields[f]]=b),!i&&d&&a.dataFilter&&(b=a.dataFilter(b,a.dataType)),i=f,f=k.shift())if("*"===f)f=i;else if("*"!==i&&i!==f){if(g=j[i+" "+f]||j["* "+f],!g)f
or(e in j)if(h=e.split(" "),h[1]===f&&(g=j[i+" "+h[0]]||j["* "+h[0]])){g===!0?g=j[e]:j[e]!==!0&&(f=h[0],k.unshift(h[1]));break}if(g!==!0)if(g&&a["throws"])b=g(b);else try{b=g(b)}catch(l){return{state:"parsererror",error:g?l:"No conversion from "+i+" to "+f}}}return{state:"success",data:b}}m.extend({active:0,lastModified:{},etag:{},ajaxSettings:{url:zc,type:"GET",isLocal:Dc.test(yc[1]),global:!0,processData:!0,async:!0,contentType:"application/x-www-form-urlencoded; charset=UTF-8",accepts:{"*":Jc,text:"text/plain",html:"text/html",xml:"application/xml, text/xml",json:"application/json, text/javascript"},contents:{xml:/xml/,html:/html/,json:/json/},responseFields:{xml:"responseXML",text:"responseText",json:"responseJSON"},converters:{"* text":String,"text html":!0,"text json":m.parseJSON,"text xml":m.parseXML},flatOptions:{url:!0,context:!0}},ajaxSetup:function(a,b){return b?Nc(Nc(a,m.ajaxSettings),b):Nc(m.ajaxSettings,a)},ajaxPrefilter:Lc(Hc),ajaxTransport:Lc(Ic),ajax:function(a,b){"
object"==typeof a&&(b=a,a=void 0),b=b||{};var c,d,e,f,g,h,i,j,k=m.ajaxSetup({},b),l=k.context||k,n=k.context&&(l.nodeType||l.jquery)?m(l):m.event,o=m.Deferred(),p=m.Callbacks("once memory"),q=k.statusCode||{},r={},s={},t=0,u="canceled",v={readyState:0,getResponseHeader:function(a){var b;if(2===t){if(!j){j={};while(b=Cc.exec(f))j[b[1].toLowerCase()]=b[2]}b=j[a.toLowerCase()]}return null==b?null:b},getAllResponseHeaders:function(){return 2===t?f:null},setRequestHeader:function(a,b){var c=a.toLowerCase();return t||(a=s[c]=s[c]||a,r[a]=b),this},overrideMimeType:function(a){return t||(k.mimeType=a),this},statusCode:function(a){var b;if(a)if(2>t)for(b in a)q[b]=[q[b],a[b]];else v.always(a[v.status]);return this},abort:function(a){var b=a||u;return i&&i.abort(b),x(0,b),this}};if(o.promise(v).complete=p.add,v.success=v.done,v.error=v.fail,k.url=((a||k.url||zc)+"").replace(Ac,"").replace(Fc,yc[1]+"//"),k.type=b.method||b.type||k.method||k.type,k.dataTypes=m.trim(k.dataType||"*").toLowerCase(
).match(E)||[""],null==k.crossDomain&&(c=Gc.exec(k.url.toLowerCase()),k.crossDomain=!(!c||c[1]===yc[1]&&c[2]===yc[2]&&(c[3]||("http:"===c[1]?"80":"443"))===(yc[3]||("http:"===yc[1]?"80":"443")))),k.data&&k.processData&&"string"!=typeof k.data&&(k.data=m.param(k.data,k.traditional)),Mc(Hc,k,b,v),2===t)return v;h=k.global,h&&0===m.active++&&m.event.trigger("ajaxStart"),k.type=k.type.toUpperCase(),k.hasContent=!Ec.test(k.type),e=k.url,k.hasContent||(k.data&&(e=k.url+=(wc.test(e)?"&":"?")+k.data,delete k.data),k.cache===!1&&(k.url=Bc.test(e)?e.replace(Bc,"$1_="+vc++):e+(wc.test(e)?"&":"?")+"_="+vc++)),k.ifModified&&(m.lastModified[e]&&v.setRequestHeader("If-Modified-Since",m.lastModified[e]),m.etag[e]&&v.setRequestHeader("If-None-Match",m.etag[e])),(k.data&&k.hasContent&&k.contentType!==!1||b.contentType)&&v.setRequestHeader("Content-Type",k.contentType),v.setRequestHeader("Accept",k.dataTypes[0]&&k.accepts[k.dataTypes[0]]?k.accepts[k.dataTypes[0]]+("*"!==k.dataTypes[0]?", "+Jc+"; q=0.0
1":""):k.accepts["*"]);for(d in k.headers)v.setRequestHeader(d,k.headers[d]);if(k.beforeSend&&(k.beforeSend.call(l,v,k)===!1||2===t))return v.abort();u="abort";for(d in{success:1,error:1,complete:1})v[d](k[d]);if(i=Mc(Ic,k,b,v)){v.readyState=1,h&&n.trigger("ajaxSend",[v,k]),k.async&&k.timeout>0&&(g=setTimeout(function(){v.abort("timeout")},k.timeout));try{t=1,i.send(r,x)}catch(w){if(!(2>t))throw w;x(-1,w)}}else x(-1,"No Transport");function x(a,b,c,d){var j,r,s,u,w,x=b;2!==t&&(t=2,g&&clearTimeout(g),i=void 0,f=d||"",v.readyState=a>0?4:0,j=a>=200&&300>a||304===a,c&&(u=Oc(k,v,c)),u=Pc(k,u,v,j),j?(k.ifModified&&(w=v.getResponseHeader("Last-Modified"),w&&(m.lastModified[e]=w),w=v.getResponseHeader("etag"),w&&(m.etag[e]=w)),204===a||"HEAD"===k.type?x="nocontent":304===a?x="notmodified":(x=u.state,r=u.data,s=u.error,j=!s)):(s=x,(a||!x)&&(x="error",0>a&&(a=0))),v.status=a,v.statusText=(b||x)+"",j?o.resolveWith(l,[r,x,v]):o.rejectWith(l,[v,x,s]),v.statusCode(q),q=void 0,h&&n.trigger(j?"ajax
Success":"ajaxError",[v,k,j?r:s]),p.fireWith(l,[v,x]),h&&(n.trigger("ajaxComplete",[v,k]),--m.active||m.event.trigger("ajaxStop")))}return v},getJSON:function(a,b,c){return m.get(a,b,c,"json")},getScript:function(a,b){return m.get(a,void 0,b,"script")}}),m.each(["get","post"],function(a,b){m[b]=function(a,c,d,e){return m.isFunction(c)&&(e=e||d,d=c,c=void 0),m.ajax({url:a,type:b,dataType:e,data:c,success:d})}}),m.each(["ajaxStart","ajaxStop","ajaxComplete","ajaxError","ajaxSuccess","ajaxSend"],function(a,b){m.fn[b]=function(a){return this.on(b,a)}}),m._evalUrl=function(a){return m.ajax({url:a,type:"GET",dataType:"script",async:!1,global:!1,"throws":!0})},m.fn.extend({wrapAll:function(a){if(m.isFunction(a))return this.each(function(b){m(this).wrapAll(a.call(this,b))});if(this[0]){var b=m(a,this[0].ownerDocument).eq(0).clone(!0);this[0].parentNode&&b.insertBefore(this[0]),b.map(function(){var a=this;while(a.firstChild&&1===a.firstChild.nodeType)a=a.firstChild;return a}).append(this)}re
turn this},wrapInner:function(a){return this.each(m.isFunction(a)?function(b){m(this).wrapInner(a.call(this,b))}:function(){var b=m(this),c=b.contents();c.length?c.wrapAll(a):b.append(a)})},wrap:function(a){var b=m.isFunction(a);return this.each(function(c){m(this).wrapAll(b?a.call(this,c):a)})},unwrap:function(){return this.parent().each(function(){m.nodeName(this,"body")||m(this).replaceWith(this.childNodes)}).end()}}),m.expr.filters.hidden=function(a){return a.offsetWidth<=0&&a.offsetHeight<=0||!k.reliableHiddenOffsets()&&"none"===(a.style&&a.style.display||m.css(a,"display"))},m.expr.filters.visible=function(a){return!m.expr.filters.hidden(a)};var Qc=/%20/g,Rc=/\[\]$/,Sc=/\r?\n/g,Tc=/^(?:submit|button|image|reset|file)$/i,Uc=/^(?:input|select|textarea|keygen)/i;function Vc(a,b,c,d){var e;if(m.isArray(b))m.each(b,function(b,e){c||Rc.test(a)?d(a,e):Vc(a+"["+("object"==typeof e?b:"")+"]",e,c,d)});else if(c||"object"!==m.type(b))d(a,b);else for(e in b)Vc(a+"["+e+"]",b[e],c,d)}m.para
m=function(a,b){var c,d=[],e=function(a,b){b=m.isFunction(b)?b():null==b?"":b,d[d.length]=encodeURIComponent(a)+"="+encodeURIComponent(b)};if(void 0===b&&(b=m.ajaxSettings&&m.ajaxSettings.traditional),m.isArray(a)||a.jquery&&!m.isPlainObject(a))m.each(a,function(){e(this.name,this.value)});else for(c in a)Vc(c,a[c],b,e);return d.join("&").replace(Qc,"+")},m.fn.extend({serialize:function(){return m.param(this.serializeArray())},serializeArray:function(){return this.map(function(){var a=m.prop(this,"elements");return a?m.makeArray(a):this}).filter(function(){var a=this.type;return this.name&&!m(this).is(":disabled")&&Uc.test(this.nodeName)&&!Tc.test(a)&&(this.checked||!W.test(a))}).map(function(a,b){var c=m(this).val();return null==c?null:m.isArray(c)?m.map(c,function(a){return{name:b.name,value:a.replace(Sc,"\r\n")}}):{name:b.name,value:c.replace(Sc,"\r\n")}}).get()}}),m.ajaxSettings.xhr=void 0!==a.ActiveXObject?function(){return!this.isLocal&&/^(get|post|head|put|delete|options)$/i.
test(this
<TRUNCATED>
[17/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/configuration.txt
----------------------------------------------------------------------
diff --git a/_sources/configuration.txt b/_sources/configuration.txt
new file mode 100644
index 0000000..3eed553
--- /dev/null
+++ b/_sources/configuration.txt
@@ -0,0 +1,230 @@
+Configuration
+-------------
+
+Setting up the sandbox in the :doc:`start` section was easy;
+building a production-grade environment requires a bit more work!
+
+Setting Configuration Options
+'''''''''''''''''''''''''''''
+
+The first time you run Airflow, it will create a file called ``airflow.cfg`` in
+your ``$AIRFLOW_HOME`` directory (``~/airflow`` by default). This file contains Airflow's configuration and you
+can edit it to change any of the settings. You can also set options with environment variables by using this format:
+``$AIRFLOW__{SECTION}__{KEY}`` (note the double underscores).
+
+For example, the
+metadata database connection string can either be set in ``airflow.cfg`` like this:
+
+.. code-block:: bash
+
+ [core]
+ sql_alchemy_conn = my_conn_string
+
+or by creating a corresponding environment variable:
+
+.. code-block:: bash
+
+ AIRFLOW__CORE__SQL_ALCHEMY_CONN=my_conn_string
+
+You can also derive the connection string at run time by appending ``_cmd`` to the key like this:
+
+.. code-block:: bash
+
+ [core]
+ sql_alchemy_conn_cmd = bash_command_to_run
+
+But only three such configuration elements namely sql_alchemy_conn, broker_url and celery_result_backend can be fetched as a command. The idea behind this is to not store passwords on boxes in plain text files. The order of precedence is as follows -
+
+1. environment variable
+2. configuration in airflow.cfg
+3. command in airflow.cfg
+4. default
+
+Setting up a Backend
+''''''''''''''''''''
+If you want to take a real test drive of Airflow, you should consider
+setting up a real database backend and switching to the LocalExecutor.
+
+As Airflow was built to interact with its metadata using the great SqlAlchemy
+library, you should be able to use any database backend supported as a
+SqlAlchemy backend. We recommend using **MySQL** or **Postgres**.
+
+.. note:: If you decide to use **Postgres**, we recommend using the ``psycopg2``
+ driver and specifying it in your SqlAlchemy connection string.
+ Also note that since SqlAlchemy does not expose a way to target a
+ specific schema in the Postgres connection URI, you may
+ want to set a default schema for your role with a
+ command similar to ``ALTER ROLE username SET search_path = airflow, foobar;``
+
+Once you've setup your database to host Airflow, you'll need to alter the
+SqlAlchemy connection string located in your configuration file
+``$AIRFLOW_HOME/airflow.cfg``. You should then also change the "executor"
+setting to use "LocalExecutor", an executor that can parallelize task
+instances locally.
+
+.. code-block:: bash
+
+ # initialize the database
+ airflow initdb
+
+Connections
+'''''''''''
+Airflow needs to know how to connect to your environment. Information
+such as hostname, port, login and passwords to other systems and services is
+handled in the ``Admin->Connection`` section of the UI. The pipeline code you
+will author will reference the 'conn_id' of the Connection objects.
+
+.. image:: img/connections.png
+
+By default, Airflow will save the passwords for the connection in plain text
+within the metadata database. The ``crypto`` package is highly recommended
+during installation. The ``crypto`` package does require that your operating
+system have libffi-dev installed.
+
+Connections in Airflow pipelines can be created using environment variables.
+The environment variable needs to have a prefix of ``AIRFLOW_CONN_`` for
+Airflow with the value in a URI format to use the connection properly. Please
+see the :doc:`concepts` documentation for more information on environment
+variables and connections.
+
+Scaling Out with Celery
+'''''''''''''''''''''''
+``CeleryExecutor`` is one of the ways you can scale out the number of workers. For this
+to work, you need to setup a Celery backend (**RabbitMQ**, **Redis**, ...) and
+change your ``airflow.cfg`` to point the executor parameter to
+``CeleryExecutor`` and provide the related Celery settings.
+
+For more information about setting up a Celery broker, refer to the
+exhaustive `Celery documentation on the topic <http://docs.celeryproject.org/en/latest/getting-started/brokers/index.html>`_.
+
+Here are a few imperative requirements for your workers:
+
+- ``airflow`` needs to be installed, and the CLI needs to be in the path
+- Airflow configuration settings should be homogeneous across the cluster
+- Operators that are executed on the worker need to have their dependencies
+ met in that context. For example, if you use the ``HiveOperator``,
+ the hive CLI needs to be installed on that box, or if you use the
+ ``MySqlOperator``, the required Python library needs to be available in
+ the ``PYTHONPATH`` somehow
+- The worker needs to have access to its ``DAGS_FOLDER``, and you need to
+ synchronize the filesystems by your own means. A common setup would be to
+ store your DAGS_FOLDER in a Git repository and sync it across machines using
+ Chef, Puppet, Ansible, or whatever you use to configure machines in your
+ environment. If all your boxes have a common mount point, having your
+ pipelines files shared there should work as well
+
+
+To kick off a worker, you need to setup Airflow and kick off the worker
+subcommand
+
+.. code-block:: bash
+
+ airflow worker
+
+Your worker should start picking up tasks as soon as they get fired in
+its direction.
+
+Note that you can also run "Celery Flower", a web UI built on top of Celery,
+to monitor your workers. You can use the shortcut command ``airflow flower``
+to start a Flower web server.
+
+
+Logs
+''''
+Users can specify a logs folder in ``airflow.cfg``. By default, it is in
+the ``AIRFLOW_HOME`` directory.
+
+In addition, users can supply a remote location for storing logs and log backups
+in cloud storage. At this time, Amazon S3 and Google Cloud Storage are supported.
+To enable this feature, ``airflow.cfg`` must be configured as in this example:
+
+.. code-block:: bash
+
+ [core]
+ # Airflow can store logs remotely in AWS S3 or Google Cloud Storage. Users
+ # must supply a remote location URL (starting with either 's3://...' or
+ # 'gs://...') and an Airflow connection id that provides access to the storage
+ # location.
+ remote_base_log_folder = s3://my-bucket/path/to/logs
+ remote_log_conn_id = MyS3Conn
+ # Use server-side encryption for logs stored in S3
+ encrypt_s3_logs = False
+
+Remote logging uses an existing Airflow connection to read/write logs. If you don't
+have a connection properly setup, this will fail. In the above example, Airflow will
+try to use ``S3Hook('MyS3Conn')``.
+
+In the Airflow Web UI, local logs take precedance over remote logs. If local logs
+can not be found or accessed, the remote logs will be displayed. Note that logs
+are only sent to remote storage once a task completes (including failure). In other
+words, remote logs for running tasks are unavailable.
+
+Scaling Out on Mesos (community contributed)
+''''''''''''''''''''''''''''''''''''''''''''
+``MesosExecutor`` allows you to schedule airflow tasks on a Mesos cluster.
+For this to work, you need a running mesos cluster and you must perform the following
+steps -
+
+1. Install airflow on a machine where web server and scheduler will run,
+ let's refer to this as the "Airflow server".
+2. On the Airflow server, install mesos python eggs from `mesos downloads <http://open.mesosphere.com/downloads/mesos/>`_.
+3. On the Airflow server, use a database (such as mysql) which can be accessed from mesos
+ slave machines and add configuration in ``airflow.cfg``.
+4. Change your ``airflow.cfg`` to point executor parameter to
+ `MesosExecutor` and provide related Mesos settings.
+5. On all mesos slaves, install airflow. Copy the ``airflow.cfg`` from
+ Airflow server (so that it uses same sql alchemy connection).
+6. On all mesos slaves, run the following for serving logs:
+
+.. code-block:: bash
+
+ airflow serve_logs
+
+7. On Airflow server, to start processing/scheduling DAGs on mesos, run:
+
+.. code-block:: bash
+
+ airflow scheduler -p
+
+Note: We need -p parameter to pickle the DAGs.
+
+You can now see the airflow framework and corresponding tasks in mesos UI.
+The logs for airflow tasks can be seen in airflow UI as usual.
+
+For more information about mesos, refer to `mesos documentation <http://mesos.apache.org/documentation/latest/>`_.
+For any queries/bugs on `MesosExecutor`, please contact `@kapil-malik <https://github.com/kapil-malik>`_.
+
+Integration with systemd
+''''''''''''''''''''''''
+Airflow can integrate with systemd based systems. This makes watching your
+daemons easy as systemd can take care of restarting a daemon on failure.
+In the ``scripts/systemd`` directory you can find unit files that
+have been tested on Redhat based systems. You can copy those to
+``/usr/lib/systemd/system``. It is assumed that Airflow will run under
+``airflow:airflow``. If not (or if you are running on a non Redhat
+based system) you probably need to adjust the unit files.
+
+Environment configuration is picked up from ``/etc/sysconfig/airflow``.
+An example file is supplied. Make sure to specify the ``SCHEDULER_RUNS``
+variable in this file when you run the scheduler. You
+can also define here, for example, ``AIRFLOW_HOME`` or ``AIRFLOW_CONFIG``.
+
+Integration with upstart
+''''''''''''''''''''''''
+Airflow can integrate with upstart based systems. Upstart automatically starts all airflow services for which you
+have a corresponding ``*.conf`` file in ``/etc/init`` upon system boot. On failure, upstart automatically restarts
+the process (until it reaches re-spawn limit set in a ``*.conf`` file).
+
+You can find sample upstart job files in the ``scripts/upstart`` directory. These files have been tested on
+Ubuntu 14.04 LTS. You may have to adjust ``start on`` and ``stop on`` stanzas to make it work on other upstart
+systems. Some of the possible options are listed in ``scripts/upstart/README``.
+
+Modify ``*.conf`` files as needed and copy to ``/etc/init`` directory. It is assumed that airflow will run
+under ``airflow:airflow``. Change ``setuid`` and ``setgid`` in ``*.conf`` files if you use other user/group
+
+You can use ``initctl`` to manually start, stop, view status of the airflow process that has been
+integrated with upstart
+
+.. code-block:: bash
+
+ initctl airflow-webserver status
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/faq.txt
----------------------------------------------------------------------
diff --git a/_sources/faq.txt b/_sources/faq.txt
new file mode 100644
index 0000000..21623fc
--- /dev/null
+++ b/_sources/faq.txt
@@ -0,0 +1,100 @@
+FAQ
+========
+
+**Why isn't my task getting scheduled?**
+
+There are very many reasons why your task might not be getting scheduled.
+Here are some of the common causes:
+
+- Does your script "compile", can the Airflow engine parse it and find your
+ DAG object. To test this, you can run ``airflow list_dags`` and
+ confirm that your DAG shows up in the list. You can also run
+ ``airflow list_tasks foo_dag_id --tree`` and confirm that your task
+ shows up in the list as expected. If you use the CeleryExecutor, you
+ may way to confirm that this works both where the scheduler runs as well
+ as where the worker runs.
+
+- Is your ``start_date`` set properly? The Airflow scheduler triggers the
+ task soon after the ``start_date + scheduler_interval`` is passed.
+
+- Is your ``start_date`` beyond where you can see it in the UI? If you
+ set your it to some time say 3 months ago, you won't be able to see
+ it in the main view in the UI, but you should be able to see it in the
+ ``Menu -> Browse ->Task Instances``.
+
+- Are the dependencies for the task met. The task instances directly
+ upstream from the task need to be in a ``success`` state. Also,
+ if you have set ``depends_on_past=True``, the previous task instance
+ needs to have succeeded (except if it is the first run for that task).
+ Also, if ``wait_for_downstream=True``, make sure you understand
+ what it means.
+ You can view how these properties are set from the ``Task Details``
+ page for your task.
+
+- Are the DagRuns you need created and active? A DagRun represents a specific
+ execution of an entire DAG and has a state (running, success, failed, ...).
+ The scheduler creates new DagRun as it moves forward, but never goes back
+ in time to create new ones. The scheduler only evaluates ``running`` DagRuns
+ to see what task instances it can trigger. Note that clearing tasks
+ instances (from the UI or CLI) does set the state of a DagRun back to
+ running. You can bulk view the list of DagRuns and alter states by clicking
+ on the schedule tag for a DAG.
+
+- Is the ``concurrency`` parameter of your DAG reached? ``concurency`` defines
+ how many ``running`` task instances a DAG is allowed to have, beyond which
+ point things get queued.
+
+- Is the ``max_active_runs`` parameter of your DAG reached? ``max_active_runs`` defines
+ how many ``running`` concurrent instances of a DAG there are allowed to be.
+
+You may also want to read the Scheduler section of the docs and make
+sure you fully understand how it proceeds.
+
+
+**How do I trigger tasks based on another task's failure?**
+
+Check out the ``Trigger Rule`` section in the Concepts section of the
+documentation
+
+**Why are connection passwords still not encrypted in the metadata db after I installed airflow[crypto]**?
+
+- Verify that the ``fernet_key`` defined in ``$AIRFLOW_HOME/airflow.cfg`` is a valid Fernet key. It must be a base64-encoded 32-byte key. You need to restart the webserver after you update the key
+- For existing connections (the ones that you had defined before installing ``airflow[crypto]`` and creating a Fernet key), you need to open each connection in the connection admin UI, re-type the password, and save it
+
+**What's the deal with ``start_date``?**
+
+``start_date`` is partly legacy from the pre-DagRun era, but it is still
+relevant in many ways. When creating a new DAG, you probably want to set
+a global ``start_date`` for your tasks using ``default_args``. The first
+DagRun to be created will be based on the ``min(start_date)`` for all your
+task. From that point on, the scheduler creates new DagRuns based on
+your ``schedule_interval`` and the corresponding task instances run as your
+dependencies are met. When introducing new tasks to your DAG, you need to
+pay special attention to ``start_date``, and may want to reactivate
+inactive DagRuns to get the new task to get onboarded properly.
+
+We recommend against using dynamic values as ``start_date``, especially
+``datetime.now()`` as it can be quite confusing. The task is triggered
+once the period closes, and in theory an ``@hourly`` DAG would never get to
+an hour after now as ``now()`` moves along.
+
+We also recommend using rounded ``start_date`` in relation to your
+``schedule_interval``. This means an ``@hourly`` would be at ``00:00``
+minutes:seconds, a ``@daily`` job at midnight, a ``@monthly`` job on the
+first of the month. You can use any sensor or a ``TimeDeltaSensor`` to delay
+the execution of tasks within that period. While ``schedule_interval``
+does allow specifying a ``datetime.timedelta``
+object, we recommend using the macros or cron expressions instead, as
+it enforces this idea of rounded schedules.
+
+When using ``depends_on_past=True`` it's important to pay special attention
+to ``start_date`` as the past dependency is not enforced only on the specific
+schedule of the ``start_date`` specified for the task. It' also
+important to watch DagRun activity status in time when introducing
+new ``depends_on_past=True``, unless you are planning on running a backfill
+for the new task(s).
+
+Also important to note is that the tasks ``start_date``, in the context of a
+backfill CLI command, get overridden by the backfill's command ``start_date``.
+This allows for a backfill on tasks that have ``depends_on_past=True`` to
+actually start, if it wasn't the case, the backfill just wouldn't start.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/index.txt
----------------------------------------------------------------------
diff --git a/_sources/index.txt b/_sources/index.txt
new file mode 100644
index 0000000..70f9355
--- /dev/null
+++ b/_sources/index.txt
@@ -0,0 +1,75 @@
+
+.. image:: img/pin_large.png
+ :width: 100
+.. image:: img/incubator.jpg
+ :width: 150
+
+Apache Airflow (incubating) Documentation
+================================
+
+Airflow is a platform to programmatically author, schedule and monitor
+workflows.
+
+Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks.
+The airflow scheduler executes your tasks on an array of workers while
+following the specified dependencies. Rich command line utilities make
+performing complex surgeries on DAGs a snap. The rich user interface
+makes it easy to visualize pipelines running in production,
+monitor progress, and troubleshoot issues when needed.
+
+When workflows are defined as code, they become more maintainable,
+versionable, testable, and collaborative.
+
+------------
+
+
+.. image:: img/airflow.gif
+
+------------
+
+Principles
+----------
+
+- **Dynamic**: Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically.
+- **Extensible**: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment.
+- **Elegant**: Airflow pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful **Jinja** templating engine.
+- **Scalable**: Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to infinity.
+
+
+Beyond the Horizon
+------------------
+
+Airflow **is not** a data streaming solution. Tasks do not move data from
+one to the other (though tasks can exchange metadata!). Airflow is not
+in the `Spark Streaming <http://spark.apache.org/streaming/>`_
+or `Storm <https://storm.apache.org/>`_ space, it is more comparable to
+`Oozie <http://oozie.apache.org/>`_ or
+`Azkaban <http://data.linkedin.com/opensource/azkaban>`_.
+
+Workflows are expected to be mostly static or slowly changing. You can think
+of the structure of the tasks in your workflow as slightly more dynamic
+than a database structure would be. Airflow workflows are expected to look
+similar from a run to the next, this allows for clarity around
+unit of work and continuity.
+
+
+Content
+-------
+.. toctree::
+ :maxdepth: 4
+
+ project
+ license
+ start
+ installation
+ tutorial
+ configuration
+ ui
+ concepts
+ profiling
+ cli
+ scheduler
+ plugins
+ security
+ faq
+ code
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/installation.txt
----------------------------------------------------------------------
diff --git a/_sources/installation.txt b/_sources/installation.txt
new file mode 100644
index 0000000..289f64f
--- /dev/null
+++ b/_sources/installation.txt
@@ -0,0 +1,90 @@
+Installation
+------------
+
+Getting Airflow
+'''''''''''''''
+
+The easiest way to install the latest stable version of Airflow is with ``pip``:
+
+.. code-block:: bash
+
+ pip install airflow
+
+You can also install Airflow with support for extra features like ``s3`` or ``postgres``:
+
+.. code-block:: bash
+
+ pip install "airflow[s3, postgres]"
+
+Extra Packages
+''''''''''''''
+
+The ``airflow`` PyPI basic package only installs what's needed to get started.
+Subpackages can be installed depending on what will be useful in your
+environment. For instance, if you don't need connectivity with Postgres,
+you won't have to go through the trouble of installing the ``postgres-devel``
+yum package, or whatever equivalent applies on the distribution you are using.
+
+Behind the scenes, Airflow does conditional imports of operators that require
+these extra dependencies.
+
+Here's the list of the subpackages and what they enable:
+
++---------------+-------------------------------------+-------------------------------------------------+
+| subpackage | install command | enables |
++===============+=====================================+=================================================+
+| all | ``pip install airflow[all]`` | All Airflow features known to man |
++---------------+-------------------------------------+-------------------------------------------------+
+| all_dbs | ``pip install airflow[all_dbs]`` | All databases integrations |
++---------------+-------------------------------------+-------------------------------------------------+
+| async | ``pip install airflow[async]`` | Async worker classes for gunicorn |
++---------------+-------------------------------------+-------------------------------------------------+
+| devel | ``pip install airflow[devel]`` | Minimum dev tools requirements |
++---------------+-------------------------------------+-------------------------------------------------+
+| devel_hadoop |``pip install airflow[devel_hadoop]``| Airflow + dependencies on the Hadoop stack |
++---------------+-------------------------------------+-------------------------------------------------+
+| celery | ``pip install airflow[celery]`` | CeleryExecutor |
++---------------+-------------------------------------+-------------------------------------------------+
+| crypto | ``pip install airflow[crypto]`` | Encrypt connection passwords in metadata db |
++---------------+-------------------------------------+-------------------------------------------------+
+| druid | ``pip install airflow[druid]`` | Druid.io related operators & hooks |
++---------------+-------------------------------------+-------------------------------------------------+
+| gcp_api | ``pip install airflow[gcp_api]`` | Google Cloud Platform hooks and operators |
+| | | (using ``google-api-python-client``) |
++---------------+-------------------------------------+-------------------------------------------------+
+| jdbc | ``pip install airflow[jdbc]`` | JDBC hooks and operators |
++---------------+-------------------------------------+-------------------------------------------------+
+| hdfs | ``pip install airflow[hdfs]`` | HDFS hooks and operators |
++---------------+-------------------------------------+-------------------------------------------------+
+| hive | ``pip install airflow[hive]`` | All Hive related operators |
++---------------+-------------------------------------+-------------------------------------------------+
+| kerberos | ``pip install airflow[kerberos]`` | kerberos integration for kerberized hadoop |
++---------------+-------------------------------------+-------------------------------------------------+
+| ldap | ``pip install airflow[ldap]`` | ldap authentication for users |
++---------------+-------------------------------------+-------------------------------------------------+
+| mssql | ``pip install airflow[mssql]`` | Microsoft SQL operators and hook, |
+| | | support as an Airflow backend |
++---------------+-------------------------------------+-------------------------------------------------+
+| mysql | ``pip install airflow[mysql]`` | MySQL operators and hook, support as |
+| | | an Airflow backend |
++---------------+-------------------------------------+-------------------------------------------------+
+| password | ``pip install airflow[password]`` | Password Authentication for users |
++---------------+-------------------------------------+-------------------------------------------------+
+| postgres | ``pip install airflow[postgres]`` | Postgres operators and hook, support |
+| | | as an Airflow backend |
++---------------+-------------------------------------+-------------------------------------------------+
+| qds | ``pip install airflow[qds]`` | Enable QDS (qubole data services) support |
++---------------+-------------------------------------+-------------------------------------------------+
+| rabbitmq | ``pip install airflow[rabbitmq]`` | Rabbitmq support as a Celery backend |
++---------------+-------------------------------------+-------------------------------------------------+
+| s3 | ``pip install airflow[s3]`` | ``S3KeySensor``, ``S3PrefixSensor`` |
++---------------+-------------------------------------+-------------------------------------------------+
+| samba | ``pip install airflow[samba]`` | ``Hive2SambaOperator`` |
++---------------+-------------------------------------+-------------------------------------------------+
+| slack | ``pip install airflow[slack]`` | ``SlackAPIPostOperator`` |
++---------------+-------------------------------------+-------------------------------------------------+
+| vertica | ``pip install airflow[vertica]`` | Vertica hook |
+| | | support as an Airflow backend |
++---------------+-------------------------------------+-------------------------------------------------+
+| cloudant | ``pip install airflow[cloudant]`` | Cloudant hook |
++---------------+-------------------------------------+-------------------------------------------------+
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/license.txt
----------------------------------------------------------------------
diff --git a/_sources/license.txt b/_sources/license.txt
new file mode 100644
index 0000000..9da26c0
--- /dev/null
+++ b/_sources/license.txt
@@ -0,0 +1,211 @@
+License
+=======
+
+.. image:: img/apache.jpg
+ :width: 150
+
+::
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright 2015 Apache Software Foundation
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+ Status API Training Shop Blog About
+ � 2016 GitHub, Inc. Terms Privacy Security Contact Help
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/plugins.txt
----------------------------------------------------------------------
diff --git a/_sources/plugins.txt b/_sources/plugins.txt
new file mode 100644
index 0000000..5dde383
--- /dev/null
+++ b/_sources/plugins.txt
@@ -0,0 +1,139 @@
+Plugins
+=======
+
+Airflow has a simple plugin manager built-in that can integrate external
+features to its core by simply dropping files in your
+``$AIRFLOW_HOME/plugins`` folder.
+
+The python modules in the ``plugins`` folder get imported,
+and **hooks**, **operators**, **macros**, **executors** and web **views**
+get integrated to Airflow's main collections and become available for use.
+
+What for?
+---------
+
+Airflow offers a generic toolbox for working with data. Different
+organizations have different stacks and different needs. Using Airflow
+plugins can be a way for companies to customize their Airflow installation
+to reflect their ecosystem.
+
+Plugins can be used as an easy way to write, share and activate new sets of
+features.
+
+There's also a need for a set of more complex applications to interact with
+different flavors of data and metadata.
+
+Examples:
+
+* A set of tools to parse Hive logs and expose Hive metadata (CPU /IO / phases/ skew /...)
+* An anomaly detection framework, allowing people to collect metrics, set thresholds and alerts
+* An auditing tool, helping understand who accesses what
+* A config-driven SLA monitoring tool, allowing you to set monitored tables and at what time
+ they should land, alert people, and expose visualizations of outages
+* ...
+
+Why build on top of Airflow?
+----------------------------
+
+Airflow has many components that can be reused when building an application:
+
+* A web server you can use to render your views
+* A metadata database to store your models
+* Access to your databases, and knowledge of how to connect to them
+* An array of workers that your application can push workload to
+* Airflow is deployed, you can just piggy back on it's deployment logistics
+* Basic charting capabilities, underlying libraries and abstractions
+
+
+Interface
+---------
+
+To create a plugin you will need to derive the
+``airflow.plugins_manager.AirflowPlugin`` class and reference the objects
+you want to plug into Airflow. Here's what the class you need to derive
+looks like:
+
+
+.. code:: python
+
+ class AirflowPlugin(object):
+ # The name of your plugin (str)
+ name = None
+ # A list of class(es) derived from BaseOperator
+ operators = []
+ # A list of class(es) derived from BaseHook
+ hooks = []
+ # A list of class(es) derived from BaseExecutor
+ executors = []
+ # A list of references to inject into the macros namespace
+ macros = []
+ # A list of objects created from a class derived
+ # from flask_admin.BaseView
+ admin_views = []
+ # A list of Blueprint object created from flask.Blueprint
+ flask_blueprints = []
+ # A list of menu links (flask_admin.base.MenuLink)
+ menu_links = []
+
+
+Example
+-------
+
+The code below defines a plugin that injects a set of dummy object
+definitions in Airflow.
+
+.. code:: python
+
+ # This is the class you derive to create a plugin
+ from airflow.plugins_manager import AirflowPlugin
+
+ from flask import Blueprint
+ from flask_admin import BaseView, expose
+ from flask_admin.base import MenuLink
+
+ # Importing base classes that we need to derive
+ from airflow.hooks.base_hook import BaseHook
+ from airflow.models import BaseOperator
+ from airflow.executors.base_executor import BaseExecutor
+
+ # Will show up under airflow.hooks.PluginHook
+ class PluginHook(BaseHook):
+ pass
+
+ # Will show up under airflow.operators.PluginOperator
+ class PluginOperator(BaseOperator):
+ pass
+
+ # Will show up under airflow.executors.PluginExecutor
+ class PluginExecutor(BaseExecutor):
+ pass
+
+ # Creating a flask admin BaseView
+ class TestView(BaseView):
+ @expose('/')
+ def test(self):
+ # in this example, put your test_plugin/test.html template at airflow/plugins/templates/test_plugin/test.html
+ return self.render("test_plugin/test.html", content="Hello galaxy!")
+ v = TestView(category="Test Plugin", name="Test View")
+
+ # Creating a flask blueprint to intergrate the templates and static folder
+ bp = Blueprint(
+ "test_plugin", __name__,
+ template_folder='templates', # registers airflow/plugins/templates as a Jinja template folder
+ static_folder='static',
+ static_url_path='/static/test_plugin')
+
+ ml = MenuLink(
+ category='Test Plugin',
+ name='Test Menu Link',
+ url='http://pythonhosted.org/airflow/')
+
+ # Defining the plugin class
+ class AirflowTestPlugin(AirflowPlugin):
+ name = "test_plugin"
+ operators = [PluginOperator]
+ flask_blueprints = [bp]
+ hooks = [PluginHook]
+ executors = [PluginExecutor]
+ admin_views = [v]
+ menu_links = [ml]
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/profiling.txt
----------------------------------------------------------------------
diff --git a/_sources/profiling.txt b/_sources/profiling.txt
new file mode 100644
index 0000000..93e6b6b
--- /dev/null
+++ b/_sources/profiling.txt
@@ -0,0 +1,39 @@
+Data Profiling
+==============
+
+Part of being productive with data is having the right weapons to
+profile the data you are working with. Airflow provides a simple query
+interface to write SQL and get results quickly, and a charting application
+letting you visualize data.
+
+Adhoc Queries
+-------------
+The adhoc query UI allows for simple SQL interactions with the database
+connections registered in Airflow.
+
+.. image:: img/adhoc.png
+
+Charts
+------
+A simple UI built on top of flask-admin and highcharts allows building
+data visualizations and charts easily. Fill in a form with a label, SQL,
+chart type, pick a source database from your environment's connectons,
+select a few other options, and save it for later use.
+
+You can even use the same templating and macros available when writing
+airflow pipelines, parameterizing your queries and modifying parameters
+directly in the URL.
+
+These charts are basic, but they're easy to create, modify and share.
+
+Chart Screenshot
+................
+
+.. image:: img/chart.png
+
+-----
+
+Chart Form Screenshot
+.....................
+
+.. image:: img/chart_form.png
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/project.txt
----------------------------------------------------------------------
diff --git a/_sources/project.txt b/_sources/project.txt
new file mode 100644
index 0000000..1e6622f
--- /dev/null
+++ b/_sources/project.txt
@@ -0,0 +1,58 @@
+Project
+=======
+
+History
+-------
+
+Airflow was started in the fall of 2014 by Maxime Beauchemin at Airbnb.
+It was open source from the very first commit and officially brought under
+the Airbnb Github and announced in the spring of 2015.
+
+The project joined the Apache Software Foundation's incubation program in the
+winter of 2016.
+
+
+Committers
+----------
+
+- @mistercrunch (Maxime "Max" Beauchemin)
+- @r39132 (Siddharth "Sid" Anand)
+- @criccomini (Chris Riccomini)
+- @bolkedebruin (Bolke de Bruin)
+- @artwr (Arthur Wiedmer)
+- @jlowin (Jeremiah Lowin)
+- @patrickleotardif (Patrick Leo Tardif)
+- @aoen (Dan Davydov)
+
+For the full list of contributors, take a look at `Airflow's Github
+Contributor page:
+<https://github.com/apache/incubator-airflow/graphs/contributors>`_
+
+
+Resources & links
+-----------------
+
+* Mailing list (send emails to
+ ``dev-subscribe@airflow.incubator.apache.org`` and
+ ``commits-subscribe@airflow.incubator.apache.org``
+ to subscribe to each)
+* `Issues <https://issues.apache.org/jira/browse/AIRFLOW>`_
+* `Airbnb Blog Post about Airflow <http://nerds.airbnb.com/airflow/>`_
+* `Airflow Common Pitfalls <https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls>`_
+* `Hadoop Summit Airflow Video <https://www.youtube.com/watch?v=oYp49mBwH60>`_
+* `Airflow at Agari Blog Post <http://agari.com/blog/airflow-agari>`_
+* `Talk: Best practices with Airflow (nov 2015) <https://youtu.be/dgaoqOZlvEA>`_
+* `Airflow Lesson 1: TriggerDagRunOperator <https://www.linkedin.com/pulse/airflow-lesson-1-triggerdagrunoperator-siddharth-anand?published=t>`_
+* `Docker Airflow (externally maintained) <https://github.com/puckel/docker-airflow>`_
+* `Airflow: Tips, Tricks, and Pitfalls @ Handy <https://medium.com/handy-tech/airflow-tips-tricks-and-pitfalls-9ba53fba14eb#.o2snqeoz7>`_
+* `Airflow Chef recipe (community contributed) <https://github.com/bahchis/airflow-cookbook>`_ ,
+ `another here <https://supermarket.chef.io/cookbooks/airflow>`_
+* Airflow Puppet Module (community contributed) `puppet-airflow <https://github.com/similarweb/puppet-airflow>`_,
+ `airflow <https://forge.puppetlabs.com/similarweb/airflow>`_
+* `Gitter (chat) Channel <https://gitter.im/airbnb/airflow>`_
+
+
+Roadmap
+-------
+
+Please refer to the Roadmap on `the wiki <https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home>`_
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/scheduler.txt
----------------------------------------------------------------------
diff --git a/_sources/scheduler.txt b/_sources/scheduler.txt
new file mode 100644
index 0000000..9c8a618
--- /dev/null
+++ b/_sources/scheduler.txt
@@ -0,0 +1,101 @@
+Scheduling & Triggers
+=====================
+
+The Airflow scheduler monitors all tasks and all DAGs, and triggers the
+task instances whose dependencies have been met. Behind the scenes,
+it monitors and stays in sync with a folder for all DAG objects it may contain,
+and periodically (every minute or so) inspects active tasks to see whether
+they can be triggered.
+
+The Airflow scheduler is designed to run as a persistent service in an
+Airflow production environment. To kick it off, all you need to do is
+execute ``airflow scheduler``. It will use the configuration specified in
+``airflow.cfg``.
+
+Note that if you run a DAG on a ``schedule_interval`` of one day,
+the run stamped ``2016-01-01`` will be trigger soon after ``2016-01-01T23:59``.
+In other words, the job instance is started once the period it covers
+has ended.
+
+The scheduler starts an instance of the executor specified in the your
+``airflow.cfg``. If it happens to be the ``LocalExecutor``, tasks will be
+executed as subprocesses; in the case of ``CeleryExecutor`` and
+``MesosExecutor``, tasks are executed remotely.
+
+To start a scheduler, simply run the command:
+
+.. code:: bash
+
+ airflow scheduler
+
+
+DAG Runs
+''''''''
+
+A DAG Run is an object representing an instantiation of the DAG in time.
+
+Each DAG may or may not have a schedule, which informs how ``DAG Runs`` are
+created. ``schedule_interval`` is defined as a DAG arguments, and receives
+preferably a
+`cron expression <https://en.wikipedia.org/wiki/Cron#CRON_expression>`_ as
+a ``str``, or a ``datetime.timedelta`` object. Alternatively, you can also
+use one of these cron "preset":
+
++--------------+----------------------------------------------------------------+---------------+
+| preset | Run once a year at midnight of January 1 | cron |
++==============+================================================================+===============+
+| ``None`` | Don't schedule, use for exclusively "externally triggered" | |
+| | DAGs | |
++--------------+----------------------------------------------------------------+---------------+
+| ``@once`` | Schedule once and only once | |
++--------------+----------------------------------------------------------------+---------------+
+| ``@hourly`` | Run once an hour at the beginning of the hour | ``0 * * * *`` |
++--------------+----------------------------------------------------------------+---------------+
+| ``@daily`` | Run once a day at midnight | ``0 0 * * *`` |
++--------------+----------------------------------------------------------------+---------------+
+| ``@weekly`` | Run once a week at midnight on Sunday morning | ``0 0 * * 0`` |
++--------------+----------------------------------------------------------------+---------------+
+| ``@monthly`` | Run once a month at midnight of the first day of the month | ``0 0 1 * *`` |
++--------------+----------------------------------------------------------------+---------------+
+| ``@yearly`` | Run once a year at midnight of January 1 | ``0 0 1 1 *`` |
++--------------+----------------------------------------------------------------+---------------+
+
+
+Your DAG will be instantiated
+for each schedule, while creating a ``DAG Run`` entry for each schedule.
+
+DAG runs have a state associated to them (running, failed, success) and
+informs the scheduler on which set of schedules should be evaluated for
+task submissions. Without the metadata at the DAG run level, the Airflow
+scheduler would have much more work to do in order to figure out what tasks
+should be triggered and come to a crawl. It might also create undesired
+processing when changing the shape of your DAG, by say adding in new
+tasks.
+
+External Triggers
+'''''''''''''''''
+
+Note that ``DAG Runs`` can also be created manually through the CLI while
+running an ``airflow trigger_dag`` command, where you can define a
+specific ``run_id``. The ``DAG Runs`` created externally to the
+scheduler get associated to the trigger's timestamp, and will be displayed
+in the UI alongside scheduled ``DAG runs``.
+
+
+To Keep in Mind
+'''''''''''''''
+* The first ``DAG Run`` is created based on the minimum ``start_date`` for the
+ tasks in your DAG.
+* Subsequent ``DAG Runs`` are created by the scheduler process, based on
+ your DAG's ``schedule_interval``, sequentially.
+* When clearing a set of tasks' state in hope of getting them to re-run,
+ it is important to keep in mind the ``DAG Run``'s state too as it defines
+ whether the scheduler should look into triggering tasks for that run.
+
+Here are some of the ways you can **unblock tasks**:
+
+* From the UI, you can **clear** (as in delete the status of) individual task instances from the task instances dialog, while defining whether you want to includes the past/future and the upstream/downstream dependencies. Note that a confirmation window comes next and allows you to see the set you are about to clear.
+* The CLI command ``airflow clear -h`` has lots of options when it comes to clearing task instance states, including specifying date ranges, targeting task_ids by specifying a regular expression, flags for including upstream and downstream relatives, and targeting task instances in specific states (``failed``, or ``success``)
+* Marking task instances as successful can be done through the UI. This is mostly to fix false negatives, or for instance when the fix has been applied outside of Airflow.
+* The ``airflow backfill`` CLI subcommand has a flag to ``--mark_success`` and allows selecting subsections of the DAG as well as specifying date ranges.
+
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/security.txt
----------------------------------------------------------------------
diff --git a/_sources/security.txt b/_sources/security.txt
new file mode 100644
index 0000000..1f33548
--- /dev/null
+++ b/_sources/security.txt
@@ -0,0 +1,249 @@
+Security
+========
+
+Web Authentication
+------------------
+
+By default, all gates are opened. An easy way to restrict access
+to the web application is to do it at the network level, or by using
+SSH tunnels.
+
+It is however possible to switch on authentication by either using one of the supplied
+backends or create your own.
+
+Password
+''''''''
+
+One of the simplest mechanisms for authentication is requiring users to specify a password before logging in.
+Password authentication requires the used of the ``password`` subpackage in your requirements file. Password hashing
+uses bcrypt before storing passwords.
+
+.. code-block:: bash
+
+ [webserver]
+ authenticate = True
+ auth_backend = airflow.contrib.auth.backends.password_auth
+
+When password auth is enabled, an initial user credential will need to be created before anyone can login. An initial
+user was not created in the migrations for this authenication backend to prevent default Airflow installations from
+attack. Creating a new user has to be done via a Python REPL on the same machine Airflow is installed.
+
+.. code-block:: bash
+
+ # navigate to the airflow installation directory
+ $ cd ~/airflow
+ $ python
+ Python 2.7.9 (default, Feb 10 2015, 03:28:08)
+ Type "help", "copyright", "credits" or "license" for more information.
+ >>> import airflow
+ >>> from airflow import models, settings
+ >>> from airflow.contrib.auth.backends.password_auth import PasswordUser
+ >>> user = PasswordUser(models.User())
+ >>> user.username = 'new_user_name'
+ >>> user.email = 'new_user_email@example.com'
+ >>> user.password = 'set_the_password'
+ >>> session = settings.Session()
+ >>> session.add(user)
+ >>> session.commit()
+ >>> session.close()
+ >>> exit()
+
+LDAP
+''''
+
+To turn on LDAP authentication configure your ``airflow.cfg`` as follows. Please note that the example uses
+an encrypted connection to the ldap server as you probably do not want passwords be readable on the network level.
+It is however possible to configure without encryption if you really want to.
+
+Additionally, if you are using Active Directory, and are not explicitly specifying an OU that your users are in,
+you will need to change ``search_scope`` to "SUBTREE".
+
+Valid search_scope options can be found in the `ldap3 Documentation <http://ldap3.readthedocs.org/searches.html?highlight=search_scope>`_
+
+.. code-block:: bash
+
+ [webserver]
+ authenticate = True
+ auth_backend = airflow.contrib.auth.backends.ldap_auth
+
+ [ldap]
+ uri = ldaps://<your.ldap.server>:<port>
+ user_filter = objectClass=*
+ user_name_attr = uid # in case of Active Directory you would use sAMAccountName
+ superuser_filter = memberOf=CN=airflow-super-users,OU=Groups,OU=RWC,OU=US,OU=NORAM,DC=example,DC=com
+ data_profiler_filter = memberOf=CN=airflow-data-profilers,OU=Groups,OU=RWC,OU=US,OU=NORAM,DC=example,DC=com
+ bind_user = cn=Manager,dc=example,dc=com
+ bind_password = insecure
+ basedn = dc=example,dc=com
+ cacert = /etc/ca/ldap_ca.crt
+ search_scope = LEVEL # Set this to SUBTREE if using Active Directory, and not specifying an Organizational Unit
+
+The superuser_filter and data_profiler_filter are optional. If defined, these configurations allow you to specify LDAP groups that users must belong to in order to have superuser (admin) and data-profiler permissions. If undefined, all users will be superusers and data profilers.
+
+Roll your own
+'''''''''''''
+
+Airflow uses ``flask_login`` and
+exposes a set of hooks in the ``airflow.default_login`` module. You can
+alter the content and make it part of the ``PYTHONPATH`` and configure it as a backend in ``airflow.cfg```.
+
+.. code-block:: bash
+
+ [webserver]
+ authenticate = True
+ auth_backend = mypackage.auth
+
+Multi-tenancy
+-------------
+
+You can filter the list of dags in webserver by owner name, when authentication
+is turned on, by setting webserver.filter_by_owner as true in your ``airflow.cfg``
+With this, when a user authenticates and logs into webserver, it will see only the dags
+which it is owner of. A super_user, will be able to see all the dags although.
+This makes the web UI a multi-tenant UI, where a user will only be able to see dags
+created by itself.
+
+
+Kerberos
+--------
+Airflow has initial support for Kerberos. This means that airflow can renew kerberos
+tickets for itself and store it in the ticket cache. The hooks and dags can make use of ticket
+to authenticate against kerberized services.
+
+Limitations
+'''''''''''
+
+Please note that at this time not all hooks have been adjusted to make use of this functionality yet.
+Also it does not integrate kerberos into the web interface and you will have to rely on network
+level security for now to make sure your service remains secure.
+
+Celery integration has not been tried and tested yet. However if you generate a key tab for every host
+and launch a ticket renewer next to every worker it will most likely work.
+
+Enabling kerberos
+'''''''''''''''''
+
+#### Airflow
+
+To enable kerberos you will need to generate a (service) key tab.
+
+.. code-block:: bash
+
+ # in the kadmin.local or kadmin shell, create the airflow principal
+ kadmin: addprinc -randkey airflow/fully.qualified.domain.name@YOUR-REALM.COM
+
+ # Create the airflow keytab file that will contain the airflow principal
+ kadmin: xst -norandkey -k airflow.keytab airflow/fully.qualified.domain.name
+
+Now store this file in a location where the airflow user can read it (chmod 600). And then add the following to
+your ``airflow.cfg``
+
+.. code-block:: bash
+
+ [core]
+ security = kerberos
+
+ [kerberos]
+ keytab = /etc/airflow/airflow.keytab
+ reinit_frequency = 3600
+ principal = airflow
+
+Launch the ticket renewer by
+
+.. code-block:: bash
+
+ # run ticket renewer
+ airflow kerberos
+
+#### Hadoop
+
+If want to use impersonation this needs to be enabled in ``core-site.xml`` of your hadoop config.
+
+.. code-block:: bash
+
+ <property>
+ <name>hadoop.proxyuser.airflow.groups</name>
+ <value>*</value>
+ </property>
+
+ <property>
+ <name>hadoop.proxyuser.airflow.users</name>
+ <value>*</value>
+ </property>
+
+ <property>
+ <name>hadoop.proxyuser.airflow.hosts</name>
+ <value>*</value>
+ </property>
+
+Of course if you need to tighten your security replace the asterisk with something more appropriate.
+
+Using kerberos authentication
+'''''''''''''''''''''''''''''
+
+The hive hook has been updated to take advantage of kerberos authentication. To allow your DAGs to use it simply
+update the connection details with, for example:
+
+.. code-block:: bash
+
+ { "use_beeline": true, "principal": "hive/_HOST@EXAMPLE.COM"}
+
+Adjust the principal to your settings. The _HOST part will be replaced by the fully qualified domain name of
+the server.
+
+You can specify if you would like to use the dag owner as the user for the connection or the user specified in the login
+section of the connection. For the login user specify the following as extra:
+
+.. code-block:: bash
+
+ { "use_beeline": true, "principal": "hive/_HOST@EXAMPLE.COM", "proxy_user": "login"}
+
+For the DAG owner use:
+
+.. code-block:: bash
+
+ { "use_beeline": true, "principal": "hive/_HOST@EXAMPLE.COM", "proxy_user": "owner"}
+
+and in your DAG, when initializing the HiveOperator, specify
+
+.. code-block:: bash
+
+ run_as_owner=True
+
+GitHub Enterprise (GHE) Authentication
+''''''''''''''''''''''''''''''''''''''
+
+The GitHub Enterprise authentication backend can be used to authenticate users
+against an installation of GitHub Enterprise using OAuth2. You can optionally
+specify a team whitelist (composed of slug cased team names) to restrict login
+to only members of those teams.
+
+*NOTE* If you do not specify a team whitelist, anyone with a valid account on
+your GHE installation will be able to login to Airflow.
+
+.. code-block:: bash
+
+ [webserver]
+ authenticate = True
+ auth_backend = airflow.contrib.auth.backends.github_enterprise_auth
+
+ [github_enterprise]
+ host = github.example.com
+ client_id = oauth_key_from_github_enterprise
+ client_secret = oauth_secret_from_github_enterprise
+ oauth_callback_route = /example/ghe_oauth/callback
+ allowed_teams = example_team_1, example_team_2
+
+Setting up GHE Authentication
+'''''''''''''''''''''''''''''
+
+An application must be setup in GHE before you can use the GHE authentication
+backend. In order to setup an application:
+
+1. Navigate to your GHE profile
+2. Select 'Applications' from the left hand nav
+3. Select the 'Developer Applications' tab
+4. Click 'Register new application'
+5. Fill in the required information (the 'Authorization callback URL' must be fully qualifed e.g. http://airflow.example.com/example/ghe_oauth/callback)
+6. Click 'Register application'
+7. Copy 'Client ID', 'Client Secret', and your callback route to your airflow.cfg according to the above example
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/start.txt
----------------------------------------------------------------------
diff --git a/_sources/start.txt b/_sources/start.txt
new file mode 100644
index 0000000..cc41d4b
--- /dev/null
+++ b/_sources/start.txt
@@ -0,0 +1,49 @@
+Quick Start
+-----------
+
+The installation is quick and straightforward.
+
+.. code-block:: bash
+
+ # airflow needs a home, ~/airflow is the default,
+ # but you can lay foundation somewhere else if you prefer
+ # (optional)
+ export AIRFLOW_HOME=~/airflow
+
+ # install from pypi using pip
+ pip install airflow
+
+ # initialize the database
+ airflow initdb
+
+ # start the web server, default port is 8080
+ airflow webserver -p 8080
+
+Upon running these commands, Airflow will create the ``$AIRFLOW_HOME`` folder
+and lay an "airflow.cfg" file with defaults that get you going fast. You can
+inspect the file either in ``$AIRFLOW_HOME/airflow.cfg``, or through the UI in
+the ``Admin->Configuration`` menu. The PID file for the webserver will be stored
+in ``$AIRFLOW_HOME/airflow-webserver.pid`` or in ``/run/airflow/webserver.pid``
+if started by systemd.
+
+Out of the box, Airflow uses a sqlite database, which you should outgrow
+fairly quickly since no parallelization is possible using this database
+backend. It works in conjunction with the ``SequentialExecutor`` which will
+only run task instances sequentially. While this is very limiting, it allows
+you to get up and running quickly and take a tour of the UI and the
+command line utilities.
+
+Here are a few commands that will trigger a few task instances. You should
+be able to see the status of the jobs change in the ``example1`` DAG as you
+run the commands below.
+
+.. code-block:: bash
+
+ # run your first task instance
+ airflow run example_bash_operator runme_0 2015-01-01
+ # run a backfill over 2 days
+ airflow backfill example_bash_operator -s 2015-01-01 -e 2015-01-02
+
+What's Next?
+''''''''''''
+From this point, you can head to the :doc:`tutorial` section for further examples or the :doc:`configuration` section if you're ready to get your hands dirty.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/tutorial.txt
----------------------------------------------------------------------
diff --git a/_sources/tutorial.txt b/_sources/tutorial.txt
new file mode 100644
index 0000000..e9d382b
--- /dev/null
+++ b/_sources/tutorial.txt
@@ -0,0 +1,429 @@
+
+Tutorial
+================
+
+This tutorial walks you through some of the fundamental Airflow concepts,
+objects, and their usage while writing your first pipeline.
+
+Example Pipeline definition
+---------------------------
+
+Here is an example of a basic pipeline definition. Do not worry if this looks
+complicated, a line by line explanation follows below.
+
+.. code:: python
+
+ """
+ Code that goes along with the Airflow tutorial located at:
+ https://github.com/airbnb/airflow/blob/master/airflow/example_dags/tutorial.py
+ """
+ from airflow import DAG
+ from airflow.operators import BashOperator
+ from datetime import datetime, timedelta
+
+
+ default_args = {
+ 'owner': 'airflow',
+ 'depends_on_past': False,
+ 'start_date': datetime(2015, 6, 1),
+ 'email': ['airflow@airflow.com'],
+ 'email_on_failure': False,
+ 'email_on_retry': False,
+ 'retries': 1,
+ 'retry_delay': timedelta(minutes=5),
+ # 'queue': 'bash_queue',
+ # 'pool': 'backfill',
+ # 'priority_weight': 10,
+ # 'end_date': datetime(2016, 1, 1),
+ }
+
+ dag = DAG('tutorial', default_args=default_args)
+
+ # t1, t2 and t3 are examples of tasks created by instatiating operators
+ t1 = BashOperator(
+ task_id='print_date',
+ bash_command='date',
+ dag=dag)
+
+ t2 = BashOperator(
+ task_id='sleep',
+ bash_command='sleep 5',
+ retries=3,
+ dag=dag)
+
+ templated_command = """
+ {% for i in range(5) %}
+ echo "{{ ds }}"
+ echo "{{ macros.ds_add(ds, 7)}}"
+ echo "{{ params.my_param }}"
+ {% endfor %}
+ """
+
+ t3 = BashOperator(
+ task_id='templated',
+ bash_command=templated_command,
+ params={'my_param': 'Parameter I passed in'},
+ dag=dag)
+
+ t2.set_upstream(t1)
+ t3.set_upstream(t1)
+
+
+It's a DAG definition file
+--------------------------
+
+One thing to wrap your head around (it may not be very intuitive for everyone
+at first) is that this Airflow Python script is really
+just a configuration file specifying the DAG's structure as code.
+The actual tasks defined here will run in a different context from
+the context of this script. Different tasks run on different workers
+at different points in time, which means that this script cannot be used
+to cross communicate between tasks. Note that for this
+purpose we have a more advanced feature called ``XCom``.
+
+People sometimes think of the DAG definition file as a place where they
+can do some actual data processing - that is not the case at all!
+The script's purpose is to define a DAG object. It needs to evaluate
+quickly (seconds, not minutes) since the scheduler will execute it
+periodically to reflect the changes if any.
+
+
+Importing Modules
+-----------------
+
+An Airflow pipeline is just a Python script that happens to define an
+Airflow DAG object. Let's start by importing the libraries we will need.
+
+.. code:: python
+
+ # The DAG object; we'll need this to instantiate a DAG
+ from airflow import DAG
+
+ # Operators; we need this to operate!
+ from airflow.operators import BashOperator
+
+Default Arguments
+-----------------
+We're about to create a DAG and some tasks, and we have the choice to
+explicitly pass a set of arguments to each task's constructor
+(which would become redundant), or (better!) we can define a dictionary
+of default parameters that we can use when creating tasks.
+
+.. code:: python
+
+ from datetime import datetime, timedelta
+
+ default_args = {
+ 'owner': 'airflow',
+ 'depends_on_past': False,
+ 'start_date': datetime(2015, 6, 1),
+ 'email': ['airflow@airflow.com'],
+ 'email_on_failure': False,
+ 'email_on_retry': False,
+ 'retries': 1,
+ 'retry_delay': timedelta(minutes=5),
+ # 'queue': 'bash_queue',
+ # 'pool': 'backfill',
+ # 'priority_weight': 10,
+ # 'end_date': datetime(2016, 1, 1),
+ }
+
+For more information about the BaseOperator's parameters and what they do,
+refer to the :py:class:``airflow.models.BaseOperator`` documentation.
+
+Also, note that you could easily define different sets of arguments that
+would serve different purposes. An example of that would be to have
+different settings between a production and development environment.
+
+
+Instantiate a DAG
+-----------------
+
+We'll need a DAG object to nest our tasks into. Here we pass a string
+that defines the ``dag_id``, which serves as a unique identifier for your DAG.
+We also pass the default argument dictionary that we just defined and
+define a ``schedule_interval`` of 1 day for the DAG.
+
+.. code:: python
+
+ dag = DAG(
+ 'tutorial', default_args=default_args, schedule_interval=timedelta(1))
+
+Tasks
+-----
+Tasks are generated when instantiating operator objects. An object
+instantiated from an operator is called a constructor. The first argument
+``task_id`` acts as a unique identifier for the task.
+
+.. code:: python
+
+ t1 = BashOperator(
+ task_id='print_date',
+ bash_command='date',
+ dag=dag)
+
+ t2 = BashOperator(
+ task_id='sleep',
+ bash_command='sleep 5',
+ retries=3,
+ dag=dag)
+
+Notice how we pass a mix of operator specific arguments (``bash_command``) and
+an argument common to all operators (``retries``) inherited
+from BaseOperator to the operator's constructor. This is simpler than
+passing every argument for every constructor call. Also, notice that in
+the second task we override the ``retries`` parameter with ``3``.
+
+The precedence rules for a task are as follows:
+
+1. Explicitly passed arguments
+2. Values that exist in the ``default_args`` dictionary
+3. The operator's default value, if one exists
+
+A task must include or inherit the arguments ``task_id`` and ``owner``,
+otherwise Airflow will raise an exception.
+
+Templating with Jinja
+---------------------
+Airflow leverages the power of
+`Jinja Templating <http://jinja.pocoo.org/docs/dev/>`_ and provides
+the pipeline author
+with a set of built-in parameters and macros. Airflow also provides
+hooks for the pipeline author to define their own parameters, macros and
+templates.
+
+This tutorial barely scratches the surface of what you can do with
+templating in Airflow, but the goal of this section is to let you know
+this feature exists, get you familiar with double curly brackets, and
+point to the most common template variable: ``{{ ds }}``.
+
+.. code:: python
+
+ templated_command = """
+ {% for i in range(5) %}
+ echo "{{ ds }}"
+ echo "{{ macros.ds_add(ds, 7) }}"
+ echo "{{ params.my_param }}"
+ {% endfor %}
+ """
+
+ t3 = BashOperator(
+ task_id='templated',
+ bash_command=templated_command,
+ params={'my_param': 'Parameter I passed in'},
+ dag=dag)
+
+Notice that the ``templated_command`` contains code logic in ``{% %}`` blocks,
+references parameters like ``{{ ds }}``, calls a function as in
+``{{ macros.ds_add(ds, 7)}}``, and references a user-defined parameter
+in ``{{ params.my_param }}``.
+
+The ``params`` hook in ``BaseOperator`` allows you to pass a dictionary of
+parameters and/or objects to your templates. Please take the time
+to understand how the parameter ``my_param`` makes it through to the template.
+
+Files can also be passed to the ``bash_command`` argument, like
+``bash_command='templated_command.sh'``, where the file location is relative to
+the directory containing the pipeline file (``tutorial.py`` in this case). This
+may be desirable for many reasons, like separating your script's logic and
+pipeline code, allowing for proper code highlighting in files composed in
+different languages, and general flexibility in structuring pipelines. It is
+also possible to define your ``template_searchpath`` as pointing to any folder
+locations in the DAG constructor call.
+
+For more information on the variables and macros that can be referenced
+in templates, make sure to read through the :ref:`macros` section
+
+Setting up Dependencies
+-----------------------
+We have two simple tasks that do not depend on each other. Here's a few ways
+you can define dependencies between them:
+
+.. code:: python
+
+ t2.set_upstream(t1)
+
+ # This means that t2 will depend on t1
+ # running successfully to run
+ # It is equivalent to
+ # t1.set_downstream(t2)
+
+ t3.set_upstream(t1)
+
+ # all of this is equivalent to
+ # dag.set_dependency('print_date', 'sleep')
+ # dag.set_dependency('print_date', 'templated')
+
+Note that when executing your script, Airflow will raise exceptions when
+it finds cycles in your DAG or when a dependency is referenced more
+than once.
+
+Recap
+-----
+Alright, so we have a pretty basic DAG. At this point your code should look
+something like this:
+
+.. code:: python
+
+ """
+ Code that goes along with the Airflow located at:
+ http://airflow.readthedocs.org/en/latest/tutorial.html
+ """
+ from airflow import DAG
+ from airflow.operators import BashOperator
+ from datetime import datetime, timedelta
+
+
+ default_args = {
+ 'owner': 'airflow',
+ 'depends_on_past': False,
+ 'start_date': datetime(2015, 6, 1),
+ 'email': ['airflow@airflow.com'],
+ 'email_on_failure': False,
+ 'email_on_retry': False,
+ 'retries': 1,
+ 'retry_delay': timedelta(minutes=5),
+ # 'queue': 'bash_queue',
+ # 'pool': 'backfill',
+ # 'priority_weight': 10,
+ # 'end_date': datetime(2016, 1, 1),
+ }
+
+ dag = DAG(
+ 'tutorial', default_args=default_args, schedule_interval=timedelta(1))
+
+ # t1, t2 and t3 are examples of tasks created by instatiating operators
+ t1 = BashOperator(
+ task_id='print_date',
+ bash_command='date',
+ dag=dag)
+
+ t2 = BashOperator(
+ task_id='sleep',
+ bash_command='sleep 5',
+ retries=3,
+ dag=dag)
+
+ templated_command = """
+ {% for i in range(5) %}
+ echo "{{ ds }}"
+ echo "{{ macros.ds_add(ds, 7)}}"
+ echo "{{ params.my_param }}"
+ {% endfor %}
+ """
+
+ t3 = BashOperator(
+ task_id='templated',
+ bash_command=templated_command,
+ params={'my_param': 'Parameter I passed in'},
+ dag=dag)
+
+ t2.set_upstream(t1)
+ t3.set_upstream(t1)
+
+Testing
+--------
+
+Running the Script
+''''''''''''''''''
+
+Time to run some tests. First let's make sure that the pipeline
+parses. Let's assume we're saving the code from the previous step in
+``tutorial.py`` in the DAGs folder referenced in your ``airflow.cfg``.
+The default location for your DAGs is ``~/airflow/dags``.
+
+.. code-block:: bash
+
+ python ~/airflow/dags/tutorial.py
+
+If the script does not raise an exception it means that you haven't done
+anything horribly wrong, and that your Airflow environment is somewhat
+sound.
+
+Command Line Metadata Validation
+'''''''''''''''''''''''''''''''''
+Let's run a few commands to validate this script further.
+
+.. code-block:: bash
+
+ # print the list of active DAGs
+ airflow list_dags
+
+ # prints the list of tasks the "tutorial" dag_id
+ airflow list_tasks tutorial
+
+ # prints the hierarchy of tasks in the tutorial DAG
+ airflow list_tasks tutorial --tree
+
+
+Testing
+'''''''
+Let's test by running the actual task instances on a specific date. The
+date specified in this context is an ``execution_date``, which simulates the
+scheduler running your task or dag at a specific date + time:
+
+.. code-block:: bash
+
+ # command layout: command subcommand dag_id task_id date
+
+ # testing print_date
+ airflow test tutorial print_date 2015-06-01
+
+ # testing sleep
+ airflow test tutorial sleep 2015-06-01
+
+Now remember what we did with templating earlier? See how this template
+gets rendered and executed by running this command:
+
+.. code-block:: bash
+
+ # testing templated
+ airflow test tutorial templated 2015-06-01
+
+This should result in displaying a verbose log of events and ultimately
+running your bash command and printing the result.
+
+Note that the ``airflow test`` command runs task instances locally, outputs
+their log to stdout (on screen), doesn't bother with dependencies, and
+doesn't communicate state (running, success, failed, ...) to the database.
+It simply allows testing a single task instance.
+
+Backfill
+''''''''
+Everything looks like it's running fine so let's run a backfill.
+``backfill`` will respect your dependencies, emit logs into files and talk to
+the database to record status. If you do have a webserver up, you'll be able
+to track the progress. ``airflow webserver`` will start a web server if you
+are interested in tracking the progress visually as your backfill progresses.
+
+Note that if you use ``depends_on_past=True``, individual task instances
+will depend on the success of the preceding task instance, except for the
+start_date specified itself, for which this dependency is disregarded.
+
+The date range in this context is a ``start_date`` and optionally an ``end_date``,
+which are used to populate the run schedule with task instances from this dag.
+
+.. code-block:: bash
+
+ # optional, start a web server in debug mode in the background
+ # airflow webserver --debug &
+
+ # start your backfill on a date range
+ airflow backfill tutorial -s 2015-06-01 -e 2015-06-07
+
+What's Next?
+-------------
+That's it, you've written, tested and backfilled your very first Airflow
+pipeline. Merging your code into a code repository that has a master scheduler
+running against it should get it to get triggered and run every day.
+
+Here's a few things you might want to do next:
+
+* Take an in-depth tour of the UI - click all the things!
+* Keep reading the docs! Especially the sections on:
+
+ * Command line interface
+ * Operators
+ * Macros
+
+* Write your first pipeline!
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_sources/ui.txt
----------------------------------------------------------------------
diff --git a/_sources/ui.txt b/_sources/ui.txt
new file mode 100644
index 0000000..4b232fa
--- /dev/null
+++ b/_sources/ui.txt
@@ -0,0 +1,102 @@
+UI / Screenshots
+=================
+The Airflow UI make it easy to monitor and troubleshoot your data pipelines.
+Here's a quick overview of some of the features and visualizations you
+can find in the Airflow UI.
+
+
+DAGs View
+.........
+List of the DAGs in your environment, and a set of shortcuts to useful pages.
+You can see exactly how many tasks succeeded, failed, or are currently
+running at a glance.
+
+------------
+
+.. image:: img/dags.png
+
+------------
+
+
+Tree View
+.........
+A tree representation of the DAG that spans across time. If a pipeline is
+late, you can quickly see where the different steps are and identify
+the blocking ones.
+
+------------
+
+.. image:: img/tree.png
+
+------------
+
+Graph View
+..........
+The graph view is perhaps the most comprehensive. Visualize your DAG's
+dependencies and their current status for a specific run.
+
+------------
+
+.. image:: img/graph.png
+
+------------
+
+Variable View
+.............
+The variable view allows you to list, create, edit or delete the key-value pair
+of a variable used during jobs. Value of a variable will be hidden if the key contains
+any words in ('password', 'secret', 'passwd', 'authorization', 'api_key', 'apikey', 'access_token')
+by default, but can be configured to show in clear-text.
+
+------------
+
+.. image:: img/variable_hidden.png
+
+------------
+
+Gantt Chart
+...........
+The Gantt chart lets you analyse task duration and overlap. You can quickly
+identify bottlenecks and where the bulk of the time is spent for specific
+DAG runs.
+
+------------
+
+.. image:: img/gantt.png
+
+------------
+
+Task Duration
+.............
+The duration of your different tasks over the past N runs. This view lets
+you find outliers and quickly understand where the time is spent in your
+DAG over many runs.
+
+
+------------
+
+.. image:: img/duration.png
+
+------------
+
+Code View
+.........
+Transparency is everything. While the code for your pipeline is in source
+control, this is a quick way to get to the code that generates the DAG and
+provide yet more context.
+
+------------
+
+.. image:: img/code.png
+
+------------
+
+Task Instance Context Menu
+..........................
+From the pages seen above (tree view, graph view, gantt, ...), it is always
+possible to click on a task instance, and get to this rich context menu
+that can take you to more detailed metadata, and perform some actions.
+
+------------
+
+.. image:: img/context.png
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/ajax-loader.gif
----------------------------------------------------------------------
diff --git a/_static/ajax-loader.gif b/_static/ajax-loader.gif
new file mode 100644
index 0000000..61faf8c
Binary files /dev/null and b/_static/ajax-loader.gif differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/apache.jpg
----------------------------------------------------------------------
diff --git a/_static/apache.jpg b/_static/apache.jpg
new file mode 100644
index 0000000..312251f
Binary files /dev/null and b/_static/apache.jpg differ
[23/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/http_operator.html
----------------------------------------------------------------------
diff --git a/_modules/http_operator.html b/_modules/http_operator.html
new file mode 100644
index 0000000..51d58eb
--- /dev/null
+++ b/_modules/http_operator.html
@@ -0,0 +1,265 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>http_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>http_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for http_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">HttpHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="SimpleHttpOperator"><a class="viewcode-back" href="../code.html#airflow.operators.SimpleHttpOperator">[docs]</a><span class="k">class</span> <span class="nc">SimpleHttpOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Calls an endpoint on an HTTP system to execute an action</span>
+
+<span class="sd"> :param http_conn_id: The connection to run the sensor against</span>
+<span class="sd"> :type http_conn_id: string</span>
+<span class="sd"> :param endpoint: The relative part of the full url</span>
+<span class="sd"> :type endpoint: string</span>
+<span class="sd"> :param method: The HTTP method to use, default = "POST"</span>
+<span class="sd"> :type method: string</span>
+<span class="sd"> :param data: The data to pass. POST-data in POST/PUT and params</span>
+<span class="sd"> in the URL for a GET request.</span>
+<span class="sd"> :type data: For POST/PUT, depends on the content-type parameter,</span>
+<span class="sd"> for GET a dictionary of key/value string pairs</span>
+<span class="sd"> :param headers: The HTTP headers to be added to the GET request</span>
+<span class="sd"> :type headers: a dictionary of string key/value pairs</span>
+<span class="sd"> :param response_check: A check against the 'requests' response object.</span>
+<span class="sd"> Returns True for 'pass' and False otherwise.</span>
+<span class="sd"> :type response_check: A lambda or defined function.</span>
+<span class="sd"> :param extra_options: Extra options for the 'requests' library, see the</span>
+<span class="sd"> 'requests' documentation (options to modify timeout, ssl, etc.)</span>
+<span class="sd"> :type extra_options: A dictionary of options, where key is string and value</span>
+<span class="sd"> depends on the option that's being modified.</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'endpoint'</span><span class="p">,</span><span class="s1">'data'</span><span class="p">,)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">()</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#f4a460'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">endpoint</span><span class="p">,</span>
+ <span class="n">method</span><span class="o">=</span><span class="s1">'POST'</span><span class="p">,</span>
+ <span class="n">data</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">headers</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">response_check</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">extra_options</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">http_conn_id</span><span class="o">=</span><span class="s1">'http_default'</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SimpleHttpOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">http_conn_id</span> <span class="o">=</span> <span class="n">http_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">=</span> <span class="n">method</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span> <span class="o">=</span> <span class="n">endpoint</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">headers</span> <span class="o">=</span> <span class="n">headers</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">data</span> <span class="o">=</span> <span class="n">data</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">response_check</span> <span class="o">=</span> <span class="n">response_check</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">extra_options</span> <span class="o">=</span> <span class="n">extra_options</span> <span class="ow">or</span> <span class="p">{}</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">http</span> <span class="o">=</span> <span class="n">HttpHook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">method</span><span class="p">,</span> <span class="n">http_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">http_conn_id</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Calling HTTP method"</span><span class="p">)</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="n">http</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">headers</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">extra_options</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">response_check</span><span class="p">:</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">response_check</span><span class="p">(</span><span class="n">response</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Response check returned False."</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/index.html
----------------------------------------------------------------------
diff --git a/_modules/index.html b/_modules/index.html
new file mode 100644
index 0000000..c3bcc45
--- /dev/null
+++ b/_modules/index.html
@@ -0,0 +1,245 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Overview: module code — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li>Overview: module code</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>All modules for which code is available</h1>
+<ul><li><a href="S3_hook.html">S3_hook</a></li>
+<li><a href="airflow/contrib/operators/hipchat_operator.html">airflow.contrib.operators.hipchat_operator</a></li>
+<li><a href="airflow/executors/celery_executor.html">airflow.executors.celery_executor</a></li>
+<li><a href="airflow/executors/local_executor.html">airflow.executors.local_executor</a></li>
+<li><a href="airflow/executors/sequential_executor.html">airflow.executors.sequential_executor</a></li>
+<li><a href="airflow/macros.html">airflow.macros</a></li>
+<ul><li><a href="airflow/macros/hive.html">airflow.macros.hive</a></li>
+</ul><li><a href="airflow/models.html">airflow.models</a></li>
+<li><a href="airflow/operators/docker_operator.html">airflow.operators.docker_operator</a></li>
+<li><a href="airflow/operators/sensors.html">airflow.operators.sensors</a></li>
+<li><a href="bash_operator.html">bash_operator</a></li>
+<li><a href="cloudant_hook.html">cloudant_hook</a></li>
+<li><a href="dagrun_operator.html">dagrun_operator</a></li>
+<li><a href="dbapi_hook.html">dbapi_hook</a></li>
+<li><a href="druid_hook.html">druid_hook</a></li>
+<li><a href="dummy_operator.html">dummy_operator</a></li>
+<li><a href="email_operator.html">email_operator</a></li>
+<li><a href="ftp_hook.html">ftp_hook</a></li>
+<li><a href="gcs_hook.html">gcs_hook</a></li>
+<li><a href="generic_transfer.html">generic_transfer</a></li>
+<li><a href="hive_hooks.html">hive_hooks</a></li>
+<li><a href="hive_operator.html">hive_operator</a></li>
+<li><a href="hive_to_druid.html">hive_to_druid</a></li>
+<li><a href="hive_to_mysql.html">hive_to_mysql</a></li>
+<li><a href="hive_to_samba_operator.html">hive_to_samba_operator</a></li>
+<li><a href="http_hook.html">http_hook</a></li>
+<li><a href="http_operator.html">http_operator</a></li>
+<li><a href="mssql_hook.html">mssql_hook</a></li>
+<li><a href="mssql_operator.html">mssql_operator</a></li>
+<li><a href="mssql_to_hive.html">mssql_to_hive</a></li>
+<li><a href="mysql_hook.html">mysql_hook</a></li>
+<li><a href="mysql_operator.html">mysql_operator</a></li>
+<li><a href="mysql_to_hive.html">mysql_to_hive</a></li>
+<li><a href="postgres_hook.html">postgres_hook</a></li>
+<li><a href="postgres_operator.html">postgres_operator</a></li>
+<li><a href="presto_check_operator.html">presto_check_operator</a></li>
+<li><a href="presto_hook.html">presto_hook</a></li>
+<li><a href="python_operator.html">python_operator</a></li>
+<li><a href="s3_to_hive_operator.html">s3_to_hive_operator</a></li>
+<li><a href="sensors.html">sensors</a></li>
+<li><a href="slack_operator.html">slack_operator</a></li>
+<li><a href="sqlite_hook.html">sqlite_hook</a></li>
+<li><a href="ssh_execute_operator.html">ssh_execute_operator</a></li>
+<li><a href="ssh_hook.html">ssh_hook</a></li>
+<li><a href="vertica_hook.html">vertica_hook</a></li>
+<li><a href="vertica_operator.html">vertica_operator</a></li>
+<li><a href="vertica_to_hive.html">vertica_to_hive</a></li>
+<li><a href="webhdfs_hook.html">webhdfs_hook</a></li>
+</ul>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/mssql_hook.html
----------------------------------------------------------------------
diff --git a/_modules/mssql_hook.html b/_modules/mssql_hook.html
new file mode 100644
index 0000000..f65b4bb
--- /dev/null
+++ b/_modules/mssql_hook.html
@@ -0,0 +1,228 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>mssql_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>mssql_hook</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for mssql_hook</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">pymssql</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks.dbapi_hook</span> <span class="kn">import</span> <span class="n">DbApiHook</span>
+
+
+<div class="viewcode-block" id="MsSqlHook"><a class="viewcode-back" href="../code.html#airflow.hooks.MsSqlHook">[docs]</a><span class="k">class</span> <span class="nc">MsSqlHook</span><span class="p">(</span><span class="n">DbApiHook</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> Interact with Microsoft SQL Server.</span>
+<span class="sd"> '''</span>
+
+ <span class="n">conn_name_attr</span> <span class="o">=</span> <span class="s1">'mssql_conn_id'</span>
+ <span class="n">default_conn_name</span> <span class="o">=</span> <span class="s1">'mssql_default'</span>
+ <span class="n">supports_autocommit</span> <span class="o">=</span> <span class="bp">True</span>
+
+<div class="viewcode-block" id="MsSqlHook.get_conn"><a class="viewcode-back" href="../code.html#airflow.hooks.MsSqlHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a mssql connection object</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mssql_conn_id</span><span class="p">)</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="n">pymssql</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">host</span><span class="p">,</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">,</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">password</span><span class="p">,</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">schema</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">conn</span></div>
+
+ <span class="k">def</span> <span class="nf">set_autocommit</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">conn</span><span class="p">,</span> <span class="n">autocommit</span><span class="p">):</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">autocommit</span><span class="p">(</span><span class="n">autocommit</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/mssql_operator.html
----------------------------------------------------------------------
diff --git a/_modules/mssql_operator.html b/_modules/mssql_operator.html
new file mode 100644
index 0000000..0226615
--- /dev/null
+++ b/_modules/mssql_operator.html
@@ -0,0 +1,234 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>mssql_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>mssql_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for mssql_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">MsSqlHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="MsSqlOperator"><a class="viewcode-back" href="../code.html#airflow.operators.MsSqlOperator">[docs]</a><span class="k">class</span> <span class="nc">MsSqlOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes sql code in a specific Microsoft SQL database</span>
+<span class="sd"> :param mssql_conn_id: reference to a specific mssql database</span>
+<span class="sd"> :type mssql_conn_id: string</span>
+<span class="sd"> :param sql: the sql code to be executed</span>
+<span class="sd"> :type sql: string or string pointing to a template file.</span>
+<span class="sd"> File must have a '.sql' extensions.</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#ededed'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="n">mssql_conn_id</span><span class="o">=</span><span class="s1">'mssql_default'</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">MsSqlOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">mssql_conn_id</span> <span class="o">=</span> <span class="n">mssql_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">parameters</span> <span class="o">=</span> <span class="n">parameters</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Executing: '</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">))</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">MsSqlHook</span><span class="p">(</span><span class="n">mssql_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mssql_conn_id</span><span class="p">)</span>
+ <span class="n">hook</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">parameters</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/mssql_to_hive.html
----------------------------------------------------------------------
diff --git a/_modules/mssql_to_hive.html b/_modules/mssql_to_hive.html
new file mode 100644
index 0000000..f5eeee9
--- /dev/null
+++ b/_modules/mssql_to_hive.html
@@ -0,0 +1,312 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>mssql_to_hive — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>mssql_to_hive</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for mssql_to_hive</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">chr</span>
+<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">OrderedDict</span>
+<span class="kn">import</span> <span class="nn">unicodecsv</span> <span class="kn">as</span> <span class="nn">csv</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">from</span> <span class="nn">tempfile</span> <span class="kn">import</span> <span class="n">NamedTemporaryFile</span>
+<span class="kn">import</span> <span class="nn">pymssql</span>
+
+
+<span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">HiveCliHook</span><span class="p">,</span> <span class="n">MsSqlHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="MsSqlToHiveTransfer"><a class="viewcode-back" href="../code.html#airflow.operators.MsSqlToHiveTransfer">[docs]</a><span class="k">class</span> <span class="nc">MsSqlToHiveTransfer</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Moves data from Microsoft SQL Server to Hive. The operator runs</span>
+<span class="sd"> your query against Microsoft SQL Server, stores the file locally</span>
+<span class="sd"> before loading it into a Hive table. If the ``create`` or</span>
+<span class="sd"> ``recreate`` arguments are set to ``True``,</span>
+<span class="sd"> a ``CREATE TABLE`` and ``DROP TABLE`` statements are generated.</span>
+<span class="sd"> Hive data types are inferred from the cursor's metadata.</span>
+<span class="sd"> Note that the table generated in Hive uses ``STORED AS textfile``</span>
+<span class="sd"> which isn't the most efficient serialization format. If a</span>
+<span class="sd"> large amount of data is loaded and/or if the table gets</span>
+<span class="sd"> queried considerably, you may want to use this operator only to</span>
+<span class="sd"> stage the data into a temporary table before loading it into its</span>
+<span class="sd"> final destination using a ``HiveOperator``.</span>
+<span class="sd"> :param sql: SQL query to execute against the Microsoft SQL Server database</span>
+<span class="sd"> :type sql: str</span>
+<span class="sd"> :param hive_table: target Hive table, use dot notation to target a</span>
+<span class="sd"> specific database</span>
+<span class="sd"> :type hive_table: str</span>
+<span class="sd"> :param create: whether to create the table if it doesn't exist</span>
+<span class="sd"> :type create: bool</span>
+<span class="sd"> :param recreate: whether to drop and recreate the table at every execution</span>
+<span class="sd"> :type recreate: bool</span>
+<span class="sd"> :param partition: target partition as a dict of partition columns and values</span>
+<span class="sd"> :type partition: dict</span>
+<span class="sd"> :param delimiter: field delimiter in the file</span>
+<span class="sd"> :type delimiter: str</span>
+<span class="sd"> :param mssql_conn_id: source Microsoft SQL Server connection</span>
+<span class="sd"> :type mssql_conn_id: str</span>
+<span class="sd"> :param hive_conn_id: destination hive connection</span>
+<span class="sd"> :type hive_conn_id: str</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,</span> <span class="s1">'partition'</span><span class="p">,</span> <span class="s1">'hive_table'</span><span class="p">)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#a0e08c'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">sql</span><span class="p">,</span>
+ <span class="n">hive_table</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="nb">chr</span><span class="p">(</span><span class="mi">1</span><span class="p">),</span>
+ <span class="n">mssql_conn_id</span><span class="o">=</span><span class="s1">'mssql_default'</span><span class="p">,</span>
+ <span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="s1">'hive_cli_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">MsSqlToHiveTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span> <span class="o">=</span> <span class="n">hive_table</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">partition</span> <span class="o">=</span> <span class="n">partition</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">create</span> <span class="o">=</span> <span class="n">create</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">recreate</span> <span class="o">=</span> <span class="n">recreate</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span> <span class="o">=</span> <span class="n">delimiter</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">mssql_conn_id</span> <span class="o">=</span> <span class="n">mssql_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span> <span class="o">=</span> <span class="n">hive_cli_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">partition</span> <span class="o">=</span> <span class="n">partition</span> <span class="ow">or</span> <span class="p">{}</span>
+
+ <span class="nd">@classmethod</span>
+ <span class="k">def</span> <span class="nf">type_map</span><span class="p">(</span><span class="n">cls</span><span class="p">,</span> <span class="n">mssql_type</span><span class="p">):</span>
+ <span class="n">t</span> <span class="o">=</span> <span class="n">pymssql</span>
+ <span class="n">d</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="n">t</span><span class="o">.</span><span class="n">BINARY</span><span class="o">.</span><span class="n">value</span><span class="p">:</span> <span class="s1">'INT'</span><span class="p">,</span>
+ <span class="n">t</span><span class="o">.</span><span class="n">DECIMAL</span><span class="o">.</span><span class="n">value</span><span class="p">:</span> <span class="s1">'FLOAT'</span><span class="p">,</span>
+ <span class="n">t</span><span class="o">.</span><span class="n">NUMBER</span><span class="o">.</span><span class="n">value</span><span class="p">:</span> <span class="s1">'INT'</span><span class="p">,</span>
+ <span class="p">}</span>
+ <span class="k">return</span> <span class="n">d</span><span class="p">[</span><span class="n">mssql_type</span><span class="p">]</span> <span class="k">if</span> <span class="n">mssql_type</span> <span class="ow">in</span> <span class="n">d</span> <span class="k">else</span> <span class="s1">'STRING'</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">hive</span> <span class="o">=</span> <span class="n">HiveCliHook</span><span class="p">(</span><span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span><span class="p">)</span>
+ <span class="n">mssql</span> <span class="o">=</span> <span class="n">MsSqlHook</span><span class="p">(</span><span class="n">mssql_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mssql_conn_id</span><span class="p">)</span>
+
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Dumping Microsoft SQL Server query results to local file"</span><span class="p">)</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="n">mssql</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">cursor</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
+ <span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+ <span class="k">with</span> <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="s2">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+ <span class="n">csv_writer</span> <span class="o">=</span> <span class="n">csv</span><span class="o">.</span><span class="n">writer</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span>
+ <span class="n">field_dict</span> <span class="o">=</span> <span class="n">OrderedDict</span><span class="p">()</span>
+ <span class="n">col_count</span> <span class="o">=</span> <span class="mi">0</span>
+ <span class="k">for</span> <span class="n">field</span> <span class="ow">in</span> <span class="n">cursor</span><span class="o">.</span><span class="n">description</span><span class="p">:</span>
+ <span class="n">col_count</span> <span class="o">+=</span> <span class="mi">1</span>
+ <span class="n">col_position</span> <span class="o">=</span> <span class="s2">"Column{position}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">position</span><span class="o">=</span><span class="n">col_count</span><span class="p">)</span>
+ <span class="n">field_dict</span><span class="p">[</span><span class="n">col_position</span> <span class="k">if</span> <span class="n">field</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="s1">''</span> <span class="k">else</span> <span class="n">field</span><span class="p">[</span><span class="mi">0</span><span class="p">]]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">type_map</span><span class="p">(</span><span class="n">field</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
+ <span class="n">csv_writer</span><span class="o">.</span><span class="n">writerows</span><span class="p">(</span><span class="n">cursor</span><span class="p">)</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
+ <span class="n">cursor</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Loading file into Hive"</span><span class="p">)</span>
+ <span class="n">hive</span><span class="o">.</span><span class="n">load_file</span><span class="p">(</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span><span class="p">,</span>
+ <span class="n">field_dict</span><span class="o">=</span><span class="n">field_dict</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">create</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">partition</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">recreate</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/mysql_hook.html
----------------------------------------------------------------------
diff --git a/_modules/mysql_hook.html b/_modules/mysql_hook.html
new file mode 100644
index 0000000..053a634
--- /dev/null
+++ b/_modules/mysql_hook.html
@@ -0,0 +1,267 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>mysql_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>mysql_hook</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for mysql_hook</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">MySQLdb</span>
+<span class="kn">import</span> <span class="nn">MySQLdb.cursors</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks.dbapi_hook</span> <span class="kn">import</span> <span class="n">DbApiHook</span>
+
+<div class="viewcode-block" id="MySqlHook"><a class="viewcode-back" href="../code.html#airflow.hooks.MySqlHook">[docs]</a><span class="k">class</span> <span class="nc">MySqlHook</span><span class="p">(</span><span class="n">DbApiHook</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> Interact with MySQL.</span>
+
+<span class="sd"> You can specify charset in the extra field of your connection</span>
+<span class="sd"> as ``{"charset": "utf8"}``. Also you can choose cursor as</span>
+<span class="sd"> ``{"cursor": "SSCursor"}``. Refer to the MySQLdb.cursors for more details.</span>
+<span class="sd"> '''</span>
+
+ <span class="n">conn_name_attr</span> <span class="o">=</span> <span class="s1">'mysql_conn_id'</span>
+ <span class="n">default_conn_name</span> <span class="o">=</span> <span class="s1">'mysql_default'</span>
+ <span class="n">supports_autocommit</span> <span class="o">=</span> <span class="bp">True</span>
+
+<div class="viewcode-block" id="MySqlHook.get_conn"><a class="viewcode-back" href="../code.html#airflow.hooks.MySqlHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a mysql connection object</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mysql_conn_id</span><span class="p">)</span>
+ <span class="n">conn_config</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s2">"user"</span><span class="p">:</span> <span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">,</span>
+ <span class="s2">"passwd"</span><span class="p">:</span> <span class="n">conn</span><span class="o">.</span><span class="n">password</span> <span class="ow">or</span> <span class="s1">''</span>
+ <span class="p">}</span>
+
+ <span class="n">conn_config</span><span class="p">[</span><span class="s2">"host"</span><span class="p">]</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">host</span> <span class="ow">or</span> <span class="s1">'localhost'</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">conn</span><span class="o">.</span><span class="n">port</span><span class="p">:</span>
+ <span class="n">conn_config</span><span class="p">[</span><span class="s2">"port"</span><span class="p">]</span> <span class="o">=</span> <span class="mi">3306</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">conn_config</span><span class="p">[</span><span class="s2">"port"</span><span class="p">]</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">port</span><span class="p">)</span>
+
+ <span class="n">conn_config</span><span class="p">[</span><span class="s2">"db"</span><span class="p">]</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">schema</span> <span class="ow">or</span> <span class="s1">''</span>
+
+ <span class="k">if</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'charset'</span><span class="p">,</span> <span class="bp">False</span><span class="p">):</span>
+ <span class="n">conn_config</span><span class="p">[</span><span class="s2">"charset"</span><span class="p">]</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="p">[</span><span class="s2">"charset"</span><span class="p">]</span>
+ <span class="k">if</span> <span class="p">(</span><span class="n">conn_config</span><span class="p">[</span><span class="s2">"charset"</span><span class="p">])</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'utf8'</span> <span class="ow">or</span>\
+ <span class="p">(</span><span class="n">conn_config</span><span class="p">[</span><span class="s2">"charset"</span><span class="p">])</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'utf-8'</span><span class="p">:</span>
+ <span class="n">conn_config</span><span class="p">[</span><span class="s2">"use_unicode"</span><span class="p">]</span> <span class="o">=</span> <span class="bp">True</span>
+ <span class="k">if</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'cursor'</span><span class="p">,</span> <span class="bp">False</span><span class="p">):</span>
+ <span class="k">if</span> <span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="p">[</span><span class="s2">"cursor"</span><span class="p">])</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'sscursor'</span><span class="p">:</span>
+ <span class="n">conn_config</span><span class="p">[</span><span class="s2">"cursorclass"</span><span class="p">]</span> <span class="o">=</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">cursors</span><span class="o">.</span><span class="n">SSCursor</span>
+ <span class="k">elif</span> <span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="p">[</span><span class="s2">"cursor"</span><span class="p">])</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'dictcursor'</span><span class="p">:</span>
+ <span class="n">conn_config</span><span class="p">[</span><span class="s2">"cursorclass"</span><span class="p">]</span> <span class="o">=</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">cursors</span><span class="o">.</span><span class="n">DictCursor</span>
+ <span class="k">elif</span> <span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="p">[</span><span class="s2">"cursor"</span><span class="p">])</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="s1">'ssdictcursor'</span><span class="p">:</span>
+ <span class="n">conn_config</span><span class="p">[</span><span class="s2">"cursorclass"</span><span class="p">]</span> <span class="o">=</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">cursors</span><span class="o">.</span><span class="n">SSDictCursor</span>
+ <span class="n">local_infile</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'local_infile'</span><span class="p">,</span><span class="bp">False</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'ssl'</span><span class="p">,</span> <span class="bp">False</span><span class="p">):</span>
+ <span class="n">conn_config</span><span class="p">[</span><span class="s1">'ssl'</span><span class="p">]</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="p">[</span><span class="s1">'ssl'</span><span class="p">]</span>
+ <span class="k">if</span> <span class="n">local_infile</span><span class="p">:</span>
+ <span class="n">conn_config</span><span class="p">[</span><span class="s2">"local_infile"</span><span class="p">]</span> <span class="o">=</span> <span class="mi">1</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="o">**</span><span class="n">conn_config</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">conn</span></div>
+
+<div class="viewcode-block" id="MySqlHook.bulk_load"><a class="viewcode-back" href="../code.html#airflow.hooks.MySqlHook.bulk_load">[docs]</a> <span class="k">def</span> <span class="nf">bulk_load</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">tmp_file</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Loads a tab-delimited file into a database table</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">cur</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">"""</span>
+<span class="s2"> LOAD DATA LOCAL INFILE '{tmp_file}'</span>
+<span class="s2"> INTO TABLE {table}</span>
+<span class="s2"> """</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="n">conn</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span></div></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[12/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/fonts/fontawesome-webfont.woff
----------------------------------------------------------------------
diff --git a/_static/fonts/fontawesome-webfont.woff b/_static/fonts/fontawesome-webfont.woff
new file mode 100644
index 0000000..628b6a5
Binary files /dev/null and b/_static/fonts/fontawesome-webfont.woff differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/incubator.jpg
----------------------------------------------------------------------
diff --git a/_static/incubator.jpg b/_static/incubator.jpg
new file mode 100644
index 0000000..6f34a85
Binary files /dev/null and b/_static/incubator.jpg differ
[21/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/presto_hook.html
----------------------------------------------------------------------
diff --git a/_modules/presto_hook.html b/_modules/presto_hook.html
new file mode 100644
index 0000000..3368d4c
--- /dev/null
+++ b/_modules/presto_hook.html
@@ -0,0 +1,298 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>presto_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>presto_hook</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for presto_hook</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">str</span>
+<span class="kn">from</span> <span class="nn">pyhive</span> <span class="kn">import</span> <span class="n">presto</span>
+<span class="kn">from</span> <span class="nn">pyhive.exc</span> <span class="kn">import</span> <span class="n">DatabaseError</span>
+
+<span class="kn">from</span> <span class="nn">airflow.hooks.dbapi_hook</span> <span class="kn">import</span> <span class="n">DbApiHook</span>
+
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s2">"pyhive"</span><span class="p">)</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
+
+
+<span class="k">class</span> <span class="nc">PrestoException</span><span class="p">(</span><span class="ne">Exception</span><span class="p">):</span>
+ <span class="k">pass</span>
+
+
+<div class="viewcode-block" id="PrestoHook"><a class="viewcode-back" href="../code.html#airflow.hooks.PrestoHook">[docs]</a><span class="k">class</span> <span class="nc">PrestoHook</span><span class="p">(</span><span class="n">DbApiHook</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Interact with Presto through PyHive!</span>
+
+<span class="sd"> >>> ph = PrestoHook()</span>
+<span class="sd"> >>> sql = "SELECT count(1) AS num FROM airflow.static_babynames"</span>
+<span class="sd"> >>> ph.get_records(sql)</span>
+<span class="sd"> [[340698]]</span>
+<span class="sd"> """</span>
+
+ <span class="n">conn_name_attr</span> <span class="o">=</span> <span class="s1">'presto_conn_id'</span>
+ <span class="n">default_conn_name</span> <span class="o">=</span> <span class="s1">'presto_default'</span>
+
+<div class="viewcode-block" id="PrestoHook.get_conn"><a class="viewcode-back" href="../code.html#airflow.hooks.PrestoHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""Returns a connection object"""</span>
+ <span class="n">db</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">presto_conn_id</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">presto</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span>
+ <span class="n">host</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">host</span><span class="p">,</span>
+ <span class="n">port</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">port</span><span class="p">,</span>
+ <span class="n">username</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">login</span><span class="p">,</span>
+ <span class="n">catalog</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'catalog'</span><span class="p">,</span> <span class="s1">'hive'</span><span class="p">),</span>
+ <span class="n">schema</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">schema</span><span class="p">)</span></div>
+
+ <span class="nd">@staticmethod</span>
+ <span class="k">def</span> <span class="nf">_strip_sql</span><span class="p">(</span><span class="n">sql</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">sql</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">rstrip</span><span class="p">(</span><span class="s1">';'</span><span class="p">)</span>
+
+<div class="viewcode-block" id="PrestoHook.get_records"><a class="viewcode-back" href="../code.html#airflow.hooks.PrestoHook.get_records">[docs]</a> <span class="k">def</span> <span class="nf">get_records</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Get a set of records from Presto</span>
+<span class="sd"> """</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">PrestoHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">get_records</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_strip_sql</span><span class="p">(</span><span class="n">hql</span><span class="p">),</span> <span class="n">parameters</span><span class="p">)</span>
+ <span class="k">except</span> <span class="n">DatabaseError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="k">if</span> <span class="p">(</span><span class="nb">hasattr</span><span class="p">(</span><span class="n">e</span><span class="p">,</span> <span class="s1">'message'</span><span class="p">)</span> <span class="ow">and</span>
+ <span class="s1">'errorName'</span> <span class="ow">in</span> <span class="n">e</span><span class="o">.</span><span class="n">message</span> <span class="ow">and</span>
+ <span class="s1">'message'</span> <span class="ow">in</span> <span class="n">e</span><span class="o">.</span><span class="n">message</span><span class="p">):</span>
+ <span class="c1"># Use the structured error data in the raised exception</span>
+ <span class="k">raise</span> <span class="n">PrestoException</span><span class="p">(</span><span class="s1">'{name}: {message}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">name</span><span class="o">=</span><span class="n">e</span><span class="o">.</span><span class="n">message</span><span class="p">[</span><span class="s1">'errorName'</span><span class="p">],</span> <span class="n">message</span><span class="o">=</span><span class="n">e</span><span class="o">.</span><span class="n">message</span><span class="p">[</span><span class="s1">'message'</span><span class="p">]))</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">PrestoException</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span></div>
+
+<div class="viewcode-block" id="PrestoHook.get_first"><a class="viewcode-back" href="../code.html#airflow.hooks.PrestoHook.get_first">[docs]</a> <span class="k">def</span> <span class="nf">get_first</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns only the first row, regardless of how many rows the query</span>
+<span class="sd"> returns.</span>
+<span class="sd"> """</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">PrestoHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">get_first</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_strip_sql</span><span class="p">(</span><span class="n">hql</span><span class="p">),</span> <span class="n">parameters</span><span class="p">)</span>
+ <span class="k">except</span> <span class="n">DatabaseError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="n">obj</span> <span class="o">=</span> <span class="nb">eval</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
+ <span class="k">raise</span> <span class="n">PrestoException</span><span class="p">(</span><span class="n">obj</span><span class="p">[</span><span class="s1">'message'</span><span class="p">])</span></div>
+
+<div class="viewcode-block" id="PrestoHook.get_pandas_df"><a class="viewcode-back" href="../code.html#airflow.hooks.PrestoHook.get_pandas_df">[docs]</a> <span class="k">def</span> <span class="nf">get_pandas_df</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Get a pandas dataframe from a sql query.</span>
+<span class="sd"> """</span>
+ <span class="kn">import</span> <span class="nn">pandas</span>
+ <span class="n">cursor</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_cursor</span><span class="p">()</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_strip_sql</span><span class="p">(</span><span class="n">hql</span><span class="p">),</span> <span class="n">parameters</span><span class="p">)</span>
+ <span class="n">data</span> <span class="o">=</span> <span class="n">cursor</span><span class="o">.</span><span class="n">fetchall</span><span class="p">()</span>
+ <span class="k">except</span> <span class="n">DatabaseError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="n">obj</span> <span class="o">=</span> <span class="nb">eval</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">))</span>
+ <span class="k">raise</span> <span class="n">PrestoException</span><span class="p">(</span><span class="n">obj</span><span class="p">[</span><span class="s1">'message'</span><span class="p">])</span>
+ <span class="n">column_descriptions</span> <span class="o">=</span> <span class="n">cursor</span><span class="o">.</span><span class="n">description</span>
+ <span class="k">if</span> <span class="n">data</span><span class="p">:</span>
+ <span class="n">df</span> <span class="o">=</span> <span class="n">pandas</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
+ <span class="n">df</span><span class="o">.</span><span class="n">columns</span> <span class="o">=</span> <span class="p">[</span><span class="n">c</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">column_descriptions</span><span class="p">]</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">df</span> <span class="o">=</span> <span class="n">pandas</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">df</span></div>
+
+<div class="viewcode-block" id="PrestoHook.run"><a class="viewcode-back" href="../code.html#airflow.hooks.PrestoHook.run">[docs]</a> <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span> <span class="n">parameters</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Execute the statement against Presto. Can be used to create views.</span>
+<span class="sd"> """</span>
+ <span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">PrestoHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_strip_sql</span><span class="p">(</span><span class="n">hql</span><span class="p">),</span> <span class="n">parameters</span><span class="p">)</span></div>
+
+ <span class="k">def</span> <span class="nf">insert_rows</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="ne">NotImplementedError</span><span class="p">()</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/python_operator.html
----------------------------------------------------------------------
diff --git a/_modules/python_operator.html b/_modules/python_operator.html
new file mode 100644
index 0000000..0a9b321
--- /dev/null
+++ b/_modules/python_operator.html
@@ -0,0 +1,338 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>python_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>python_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for python_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">str</span>
+<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span><span class="p">,</span> <span class="n">TaskInstance</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.state</span> <span class="kn">import</span> <span class="n">State</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="kn">import</span> <span class="n">settings</span>
+
+
+<div class="viewcode-block" id="PythonOperator"><a class="viewcode-back" href="../code.html#airflow.operators.PythonOperator">[docs]</a><span class="k">class</span> <span class="nc">PythonOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes a Python callable</span>
+
+<span class="sd"> :param python_callable: A reference to an object that is callable</span>
+<span class="sd"> :type python_callable: python callable</span>
+<span class="sd"> :param op_kwargs: a dictionary of keyword arguments that will get unpacked</span>
+<span class="sd"> in your function</span>
+<span class="sd"> :type op_kwargs: dict</span>
+<span class="sd"> :param op_args: a list of positional arguments that will get unpacked when</span>
+<span class="sd"> calling your callable</span>
+<span class="sd"> :type op_args: list</span>
+<span class="sd"> :param provide_context: if set to true, Airflow will pass a set of</span>
+<span class="sd"> keyword arguments that can be used in your function. This set of</span>
+<span class="sd"> kwargs correspond exactly to what you can use in your jinja</span>
+<span class="sd"> templates. For this to work, you need to define `**kwargs` in your</span>
+<span class="sd"> function header.</span>
+<span class="sd"> :type provide_context: bool</span>
+<span class="sd"> :param templates_dict: a dictionary where the values are templates that</span>
+<span class="sd"> will get templated by the Airflow engine sometime between</span>
+<span class="sd"> ``__init__`` and ``execute`` takes place and are made available</span>
+<span class="sd"> in your callable's context after the template has been applied</span>
+<span class="sd"> :type templates_dict: dict of str</span>
+<span class="sd"> :param templates_exts: a list of file extensions to resolve while</span>
+<span class="sd"> processing templated fields, for examples ``['.sql', '.hql']``</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'templates_dict'</span><span class="p">,)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">()</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#ffefeb'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">python_callable</span><span class="p">,</span>
+ <span class="n">op_args</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">op_kwargs</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">provide_context</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">templates_dict</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">templates_exts</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">PythonOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">python_callable</span> <span class="o">=</span> <span class="n">python_callable</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">op_args</span> <span class="o">=</span> <span class="n">op_args</span> <span class="ow">or</span> <span class="p">[]</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">op_kwargs</span> <span class="o">=</span> <span class="n">op_kwargs</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">provide_context</span> <span class="o">=</span> <span class="n">provide_context</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">templates_dict</span> <span class="o">=</span> <span class="n">templates_dict</span>
+ <span class="k">if</span> <span class="n">templates_exts</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">template_ext</span> <span class="o">=</span> <span class="n">templates_exts</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">provide_context</span><span class="p">:</span>
+ <span class="n">context</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">op_kwargs</span><span class="p">)</span>
+ <span class="n">context</span><span class="p">[</span><span class="s1">'templates_dict'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">templates_dict</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">op_kwargs</span> <span class="o">=</span> <span class="n">context</span>
+
+ <span class="n">return_value</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">python_callable</span><span class="p">(</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">op_args</span><span class="p">,</span> <span class="o">**</span><span class="bp">self</span><span class="o">.</span><span class="n">op_kwargs</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Done. Returned value was: "</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">return_value</span><span class="p">))</span>
+ <span class="k">return</span> <span class="n">return_value</span></div>
+
+
+<div class="viewcode-block" id="BranchPythonOperator"><a class="viewcode-back" href="../code.html#airflow.operators.BranchPythonOperator">[docs]</a><span class="k">class</span> <span class="nc">BranchPythonOperator</span><span class="p">(</span><span class="n">PythonOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Allows a workflow to "branch" or follow a single path following the</span>
+<span class="sd"> execution of this task.</span>
+
+<span class="sd"> It derives the PythonOperator and expects a Python function that returns</span>
+<span class="sd"> the task_id to follow. The task_id returned should point to a task</span>
+<span class="sd"> directly downstream from {self}. All other "branches" or</span>
+<span class="sd"> directly downstream tasks are marked with a state of ``skipped`` so that</span>
+<span class="sd"> these paths can't move forward. The ``skipped`` states are propageted</span>
+<span class="sd"> downstream to allow for the DAG state to fill up and the DAG run's state</span>
+<span class="sd"> to be inferred.</span>
+
+<span class="sd"> Note that using tasks with ``depends_on_past=True`` downstream from</span>
+<span class="sd"> ``BranchPythonOperator`` is logically unsound as ``skipped`` status</span>
+<span class="sd"> will invariably lead to block tasks that depend on their past successes.</span>
+<span class="sd"> ``skipped`` states propagates where all directly upstream tasks are</span>
+<span class="sd"> ``skipped``.</span>
+<span class="sd"> """</span>
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">branch</span> <span class="o">=</span> <span class="nb">super</span><span class="p">(</span><span class="n">BranchPythonOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">context</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Following branch "</span> <span class="o">+</span> <span class="n">branch</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Marking other directly downstream tasks as skipped"</span><span class="p">)</span>
+ <span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
+ <span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="n">context</span><span class="p">[</span><span class="s1">'task'</span><span class="p">]</span><span class="o">.</span><span class="n">downstream_list</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">task</span><span class="o">.</span><span class="n">task_id</span> <span class="o">!=</span> <span class="n">branch</span><span class="p">:</span>
+ <span class="n">ti</span> <span class="o">=</span> <span class="n">TaskInstance</span><span class="p">(</span>
+ <span class="n">task</span><span class="p">,</span> <span class="n">execution_date</span><span class="o">=</span><span class="n">context</span><span class="p">[</span><span class="s1">'ti'</span><span class="p">]</span><span class="o">.</span><span class="n">execution_date</span><span class="p">)</span>
+ <span class="n">ti</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">State</span><span class="o">.</span><span class="n">SKIPPED</span>
+ <span class="n">ti</span><span class="o">.</span><span class="n">start_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+ <span class="n">ti</span><span class="o">.</span><span class="n">end_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span><span class="n">ti</span><span class="p">)</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Done."</span><span class="p">)</span></div>
+
+
+<div class="viewcode-block" id="ShortCircuitOperator"><a class="viewcode-back" href="../code.html#airflow.operators.ShortCircuitOperator">[docs]</a><span class="k">class</span> <span class="nc">ShortCircuitOperator</span><span class="p">(</span><span class="n">PythonOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Allows a workflow to continue only if a condition is met. Otherwise, the</span>
+<span class="sd"> workflow "short-circuits" and downstream tasks are skipped.</span>
+
+<span class="sd"> The ShortCircuitOperator is derived from the PythonOperator. It evaluates a</span>
+<span class="sd"> condition and short-circuits the workflow if the condition is False. Any</span>
+<span class="sd"> downstream tasks are marked with a state of "skipped". If the condition is</span>
+<span class="sd"> True, downstream tasks proceed as normal.</span>
+
+<span class="sd"> The condition is determined by the result of `python_callable`.</span>
+<span class="sd"> """</span>
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">condition</span> <span class="o">=</span> <span class="nb">super</span><span class="p">(</span><span class="n">ShortCircuitOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">context</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Condition result is {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">condition</span><span class="p">))</span>
+ <span class="k">if</span> <span class="n">condition</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Proceeding with downstream tasks...'</span><span class="p">)</span>
+ <span class="k">return</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Skipping downstream tasks...'</span><span class="p">)</span>
+ <span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
+ <span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="n">context</span><span class="p">[</span><span class="s1">'task'</span><span class="p">]</span><span class="o">.</span><span class="n">downstream_list</span><span class="p">:</span>
+ <span class="n">ti</span> <span class="o">=</span> <span class="n">TaskInstance</span><span class="p">(</span>
+ <span class="n">task</span><span class="p">,</span> <span class="n">execution_date</span><span class="o">=</span><span class="n">context</span><span class="p">[</span><span class="s1">'ti'</span><span class="p">]</span><span class="o">.</span><span class="n">execution_date</span><span class="p">)</span>
+ <span class="n">ti</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">State</span><span class="o">.</span><span class="n">SKIPPED</span>
+ <span class="n">ti</span><span class="o">.</span><span class="n">start_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+ <span class="n">ti</span><span class="o">.</span><span class="n">end_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span><span class="n">ti</span><span class="p">)</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Done."</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/s3_to_hive_operator.html
----------------------------------------------------------------------
diff --git a/_modules/s3_to_hive_operator.html b/_modules/s3_to_hive_operator.html
new file mode 100644
index 0000000..e0016ac
--- /dev/null
+++ b/_modules/s3_to_hive_operator.html
@@ -0,0 +1,353 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>s3_to_hive_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>s3_to_hive_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for s3_to_hive_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">next</span>
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">zip</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">from</span> <span class="nn">tempfile</span> <span class="kn">import</span> <span class="n">NamedTemporaryFile</span>
+
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">HiveCliHook</span><span class="p">,</span> <span class="n">S3Hook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="S3ToHiveTransfer"><a class="viewcode-back" href="../code.html#airflow.operators.S3ToHiveTransfer">[docs]</a><span class="k">class</span> <span class="nc">S3ToHiveTransfer</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Moves data from S3 to Hive. The operator downloads a file from S3,</span>
+<span class="sd"> stores the file locally before loading it into a Hive table.</span>
+<span class="sd"> If the ``create`` or ``recreate`` arguments are set to ``True``,</span>
+<span class="sd"> a ``CREATE TABLE`` and ``DROP TABLE`` statements are generated.</span>
+<span class="sd"> Hive data types are inferred from the cursor's metadata from.</span>
+
+<span class="sd"> Note that the table generated in Hive uses ``STORED AS textfile``</span>
+<span class="sd"> which isn't the most efficient serialization format. If a</span>
+<span class="sd"> large amount of data is loaded and/or if the tables gets</span>
+<span class="sd"> queried considerably, you may want to use this operator only to</span>
+<span class="sd"> stage the data into a temporary table before loading it into its</span>
+<span class="sd"> final destination using a ``HiveOperator``.</span>
+
+<span class="sd"> :param s3_key: The key to be retrieved from S3</span>
+<span class="sd"> :type s3_key: str</span>
+<span class="sd"> :param field_dict: A dictionary of the fields name in the file</span>
+<span class="sd"> as keys and their Hive types as values</span>
+<span class="sd"> :type field_dict: dict</span>
+<span class="sd"> :param hive_table: target Hive table, use dot notation to target a</span>
+<span class="sd"> specific database</span>
+<span class="sd"> :type hive_table: str</span>
+<span class="sd"> :param create: whether to create the table if it doesn't exist</span>
+<span class="sd"> :type create: bool</span>
+<span class="sd"> :param recreate: whether to drop and recreate the table at every</span>
+<span class="sd"> execution</span>
+<span class="sd"> :type recreate: bool</span>
+<span class="sd"> :param partition: target partition as a dict of partition columns</span>
+<span class="sd"> and values</span>
+<span class="sd"> :type partition: dict</span>
+<span class="sd"> :param headers: whether the file contains column names on the first</span>
+<span class="sd"> line</span>
+<span class="sd"> :type headers: bool</span>
+<span class="sd"> :param check_headers: whether the column names on the first line should be</span>
+<span class="sd"> checked against the keys of field_dict</span>
+<span class="sd"> :type check_headers: bool</span>
+<span class="sd"> :param wildcard_match: whether the s3_key should be interpreted as a Unix</span>
+<span class="sd"> wildcard pattern</span>
+<span class="sd"> :type wildcard_match: bool</span>
+<span class="sd"> :param delimiter: field delimiter in the file</span>
+<span class="sd"> :type delimiter: str</span>
+<span class="sd"> :param s3_conn_id: source s3 connection</span>
+<span class="sd"> :type s3_conn_id: str</span>
+<span class="sd"> :param hive_conn_id: destination hive connection</span>
+<span class="sd"> :type hive_conn_id: str</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'s3_key'</span><span class="p">,</span> <span class="s1">'partition'</span><span class="p">,</span> <span class="s1">'hive_table'</span><span class="p">)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">()</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#a0e08c'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">s3_key</span><span class="p">,</span>
+ <span class="n">field_dict</span><span class="p">,</span>
+ <span class="n">hive_table</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="s1">','</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">headers</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">check_headers</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">wildcard_match</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">s3_conn_id</span><span class="o">=</span><span class="s1">'s3_default'</span><span class="p">,</span>
+ <span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="s1">'hive_cli_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">S3ToHiveTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span> <span class="o">=</span> <span class="n">s3_key</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">field_dict</span> <span class="o">=</span> <span class="n">field_dict</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span> <span class="o">=</span> <span class="n">hive_table</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span> <span class="o">=</span> <span class="n">delimiter</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">create</span> <span class="o">=</span> <span class="n">create</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">recreate</span> <span class="o">=</span> <span class="n">recreate</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">partition</span> <span class="o">=</span> <span class="n">partition</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">headers</span> <span class="o">=</span> <span class="n">headers</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">check_headers</span> <span class="o">=</span> <span class="n">check_headers</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">wildcard_match</span> <span class="o">=</span> <span class="n">wildcard_match</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span> <span class="o">=</span> <span class="n">hive_cli_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span> <span class="o">=</span> <span class="n">s3_conn_id</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive</span> <span class="o">=</span> <span class="n">HiveCliHook</span><span class="p">(</span><span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3</span> <span class="o">=</span> <span class="n">S3Hook</span><span class="p">(</span><span class="n">s3_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Downloading S3 file"</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">wildcard_match</span><span class="p">:</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">check_for_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"No key matches {0}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">))</span>
+ <span class="n">s3_key_object</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">get_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">check_for_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">):</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span>
+ <span class="s2">"The key {0} does not exists"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">))</span>
+ <span class="n">s3_key_object</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_key</span><span class="p">)</span>
+ <span class="k">with</span> <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="s2">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Dumping S3 key {0} contents to local"</span>
+ <span class="s2">" file {1}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">s3_key_object</span><span class="o">.</span><span class="n">key</span><span class="p">,</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">))</span>
+ <span class="n">s3_key_object</span><span class="o">.</span><span class="n">get_contents_to_file</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">headers</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Loading file into Hive"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive</span><span class="o">.</span><span class="n">load_file</span><span class="p">(</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span><span class="p">,</span>
+ <span class="n">field_dict</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">field_dict</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">create</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">partition</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">recreate</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span> <span class="k">as</span> <span class="n">tmpf</span><span class="p">:</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">check_headers</span><span class="p">:</span>
+ <span class="n">header_l</span> <span class="o">=</span> <span class="n">tmpf</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span>
+ <span class="n">header_line</span> <span class="o">=</span> <span class="n">header_l</span><span class="o">.</span><span class="n">rstrip</span><span class="p">()</span>
+ <span class="n">header_list</span> <span class="o">=</span> <span class="n">header_line</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">)</span>
+ <span class="n">field_names</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">field_dict</span><span class="o">.</span><span class="n">keys</span><span class="p">())</span>
+ <span class="n">test_field_match</span> <span class="o">=</span> <span class="p">[</span><span class="n">h1</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="o">==</span> <span class="n">h2</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span> <span class="k">for</span> <span class="n">h1</span><span class="p">,</span> <span class="n">h2</span>
+ <span class="ow">in</span> <span class="nb">zip</span><span class="p">(</span><span class="n">header_list</span><span class="p">,</span> <span class="n">field_names</span><span class="p">)]</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">(</span><span class="n">test_field_match</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"Headers do not match field names"</span>
+ <span class="s2">"File headers:</span><span class="se">\n</span><span class="s2"> {header_list}</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="s2">"Field names: </span><span class="se">\n</span><span class="s2"> {field_names}</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="s2">""</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Headers do not match the "</span>
+ <span class="s2">"field_dict keys"</span><span class="p">)</span>
+ <span class="k">with</span> <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="s2">"w"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f_no_headers</span><span class="p">:</span>
+ <span class="n">tmpf</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
+ <span class="nb">next</span><span class="p">(</span><span class="n">tmpf</span><span class="p">)</span>
+ <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">tmpf</span><span class="p">:</span>
+ <span class="n">f_no_headers</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
+ <span class="n">f_no_headers</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Loading file without headers into Hive"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive</span><span class="o">.</span><span class="n">load_file</span><span class="p">(</span>
+ <span class="n">f_no_headers</span><span class="o">.</span><span class="n">name</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_table</span><span class="p">,</span>
+ <span class="n">field_dict</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">field_dict</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">create</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">partition</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">recreate</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[33/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/airflow/contrib/operators/hipchat_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/contrib/operators/hipchat_operator.html b/_modules/airflow/contrib/operators/hipchat_operator.html
new file mode 100644
index 0000000..370d812
--- /dev/null
+++ b/_modules/airflow/contrib/operators/hipchat_operator.html
@@ -0,0 +1,330 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.contrib.operators.hipchat_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../../../../index.html"/>
+ <link rel="up" title="Module code" href="../../../index.html"/>
+
+
+ <script src="../../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../../index.html">Module code</a> »</li>
+
+ <li>airflow.contrib.operators.hipchat_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.contrib.operators.hipchat_operator</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">import</span> <span class="nn">requests</span>
+<span class="kn">import</span> <span class="nn">json</span>
+
+
+<div class="viewcode-block" id="HipChatAPIOperator"><a class="viewcode-back" href="../../../../code.html#airflow.contrib.operators.hipchat_operator.HipChatAPIOperator">[docs]</a><span class="k">class</span> <span class="nc">HipChatAPIOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Base HipChat Operator.</span>
+<span class="sd"> All derived HipChat operators reference from HipChat's official REST API documentation</span>
+<span class="sd"> at https://www.hipchat.com/docs/apiv2. Before using any HipChat API operators you need</span>
+<span class="sd"> to get an authentication token at https://www.hipchat.com/docs/apiv2/auth.</span>
+<span class="sd"> In the future additional HipChat operators will be derived from this class as well.</span>
+
+<span class="sd"> :param token: HipChat REST API authentication token</span>
+<span class="sd"> :type token: str</span>
+<span class="sd"> :param base_url: HipChat REST API base url.</span>
+<span class="sd"> :type base_url: str</span>
+<span class="sd"> """</span>
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">token</span><span class="p">,</span>
+ <span class="n">base_url</span><span class="o">=</span><span class="s1">'https://api.hipchat.com/v2'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HipChatAPIOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">token</span> <span class="o">=</span> <span class="n">token</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">base_url</span> <span class="o">=</span> <span class="n">base_url</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">=</span> <span class="bp">None</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">url</span> <span class="o">=</span> <span class="bp">None</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">body</span> <span class="o">=</span> <span class="bp">None</span>
+
+ <span class="k">def</span> <span class="nf">prepare_request</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Used by the execute function. Set the request method, url, and body of HipChat's</span>
+<span class="sd"> REST API call.</span>
+
+<span class="sd"> Override in child class. Each HipChatAPI child operator is responsible for having</span>
+<span class="sd"> a prepare_request method call which sets self.method, self.url, and self.body.</span>
+<span class="sd"> """</span>
+ <span class="k">pass</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">prepare_request</span><span class="p">()</span>
+
+ <span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">request</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">method</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">url</span><span class="p">,</span>
+ <span class="n">headers</span><span class="o">=</span><span class="p">{</span>
+ <span class="s1">'Content-Type'</span><span class="p">:</span> <span class="s1">'application/json'</span><span class="p">,</span>
+ <span class="s1">'Authorization'</span><span class="p">:</span> <span class="s1">'Bearer </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span> <span class="bp">self</span><span class="o">.</span><span class="n">token</span><span class="p">},</span>
+ <span class="n">data</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">body</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">response</span><span class="o">.</span><span class="n">status_code</span> <span class="o">>=</span> <span class="mi">400</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'HipChat API call failed: </span><span class="si">%s</span><span class="s1"> </span><span class="si">%s</span><span class="s1">'</span><span class="p">,</span>
+ <span class="n">response</span><span class="o">.</span><span class="n">status_code</span><span class="p">,</span> <span class="n">response</span><span class="o">.</span><span class="n">reason</span><span class="p">)</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'HipChat API call failed: </span><span class="si">%s</span><span class="s1"> </span><span class="si">%s</span><span class="s1">'</span> <span class="o">%</span>
+ <span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">status_code</span><span class="p">,</span> <span class="n">response</span><span class="o">.</span><span class="n">reason</span><span class="p">))</span></div>
+
+
+<div class="viewcode-block" id="HipChatAPISendRoomNotificationOperator"><a class="viewcode-back" href="../../../../code.html#airflow.contrib.operators.hipchat_operator.HipChatAPISendRoomNotificationOperator">[docs]</a><span class="k">class</span> <span class="nc">HipChatAPISendRoomNotificationOperator</span><span class="p">(</span><span class="n">HipChatAPIOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Send notification to a specific HipChat room.</span>
+<span class="sd"> More info: https://www.hipchat.com/docs/apiv2/method/send_room_notification</span>
+
+<span class="sd"> :param room_id: Room in which to send notification on HipChat</span>
+<span class="sd"> :type room_id: str</span>
+<span class="sd"> :param message: The message body</span>
+<span class="sd"> :type message: str</span>
+<span class="sd"> :param frm: Label to be shown in addition to sender's name</span>
+<span class="sd"> :type frm: str</span>
+<span class="sd"> :param message_format: How the notification is rendered: html or text</span>
+<span class="sd"> :type message_format: str</span>
+<span class="sd"> :param color: Background color of the msg: yellow, green, red, purple, gray, or random</span>
+<span class="sd"> :type color: str</span>
+<span class="sd"> :param attach_to: The message id to attach this notification to</span>
+<span class="sd"> :type attach_to: str</span>
+<span class="sd"> :param notify: Whether this message should trigger a user notification</span>
+<span class="sd"> :type notify: bool</span>
+<span class="sd"> :param card: HipChat-defined card object</span>
+<span class="sd"> :type card: dict</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'token'</span><span class="p">,</span> <span class="s1">'room_id'</span><span class="p">,</span> <span class="s1">'message'</span><span class="p">)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#2980b9'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">room_id</span><span class="p">,</span> <span class="n">message</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HipChatAPISendRoomNotificationOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">room_id</span> <span class="o">=</span> <span class="n">room_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">message</span> <span class="o">=</span> <span class="n">message</span>
+ <span class="n">default_options</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'message_format'</span><span class="p">:</span> <span class="s1">'html'</span><span class="p">,</span>
+ <span class="s1">'color'</span><span class="p">:</span> <span class="s1">'yellow'</span><span class="p">,</span>
+ <span class="s1">'frm'</span><span class="p">:</span> <span class="s1">'airflow'</span><span class="p">,</span>
+ <span class="s1">'attach_to'</span><span class="p">:</span> <span class="bp">None</span><span class="p">,</span>
+ <span class="s1">'notify'</span><span class="p">:</span> <span class="bp">False</span><span class="p">,</span>
+ <span class="s1">'card'</span><span class="p">:</span> <span class="bp">None</span>
+ <span class="p">}</span>
+ <span class="k">for</span> <span class="p">(</span><span class="n">prop</span><span class="p">,</span> <span class="n">default</span><span class="p">)</span> <span class="ow">in</span> <span class="n">default_options</span><span class="o">.</span><span class="n">iteritems</span><span class="p">():</span>
+ <span class="nb">setattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">prop</span><span class="p">,</span> <span class="n">kwargs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">prop</span><span class="p">,</span> <span class="n">default</span><span class="p">))</span>
+
+ <span class="k">def</span> <span class="nf">prepare_request</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">params</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'message'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">message</span><span class="p">,</span>
+ <span class="s1">'message_format'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">message_format</span><span class="p">,</span>
+ <span class="s1">'color'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">color</span><span class="p">,</span>
+ <span class="s1">'from'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">frm</span><span class="p">,</span>
+ <span class="s1">'attach_to'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">attach_to</span><span class="p">,</span>
+ <span class="s1">'notify'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">notify</span><span class="p">,</span>
+ <span class="s1">'card'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">card</span>
+ <span class="p">}</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">method</span> <span class="o">=</span> <span class="s1">'POST'</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">url</span> <span class="o">=</span> <span class="s1">'</span><span class="si">%s</span><span class="s1">/room/</span><span class="si">%s</span><span class="s1">/notification'</span> <span class="o">%</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">base_url</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">room_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">body</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="nb">dict</span><span class="p">(</span>
+ <span class="p">(</span><span class="n">k</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">),</span> <span class="n">v</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">))</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">params</span><span class="o">.</span><span class="n">iteritems</span><span class="p">()</span> <span class="k">if</span> <span class="n">v</span><span class="p">))</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/airflow/executors/celery_executor.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/executors/celery_executor.html b/_modules/airflow/executors/celery_executor.html
new file mode 100644
index 0000000..4f91594
--- /dev/null
+++ b/_modules/airflow/executors/celery_executor.html
@@ -0,0 +1,298 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.executors.celery_executor — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../../../index.html"/>
+ <link rel="up" title="Module code" href="../../index.html"/>
+
+
+ <script src="../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../index.html">Module code</a> »</li>
+
+ <li>airflow.executors.celery_executor</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.executors.celery_executor</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">object</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">import</span> <span class="nn">subprocess</span>
+<span class="kn">import</span> <span class="nn">time</span>
+
+<span class="kn">from</span> <span class="nn">celery</span> <span class="kn">import</span> <span class="n">Celery</span>
+<span class="kn">from</span> <span class="nn">celery</span> <span class="kn">import</span> <span class="n">states</span> <span class="k">as</span> <span class="n">celery_states</span>
+
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.executors.base_executor</span> <span class="kn">import</span> <span class="n">BaseExecutor</span>
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="kn">import</span> <span class="n">configuration</span>
+
+<span class="n">PARALLELISM</span> <span class="o">=</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'PARALLELISM'</span><span class="p">)</span>
+
+<span class="sd">'''</span>
+<span class="sd">To start the celery worker, run the command:</span>
+<span class="sd">airflow worker</span>
+<span class="sd">'''</span>
+
+<span class="n">DEFAULT_QUEUE</span> <span class="o">=</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'celery'</span><span class="p">,</span> <span class="s1">'DEFAULT_QUEUE'</span><span class="p">)</span>
+
+
+<span class="k">class</span> <span class="nc">CeleryConfig</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+ <span class="n">CELERY_ACCEPT_CONTENT</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'json'</span><span class="p">,</span> <span class="s1">'pickle'</span><span class="p">]</span>
+ <span class="n">CELERYD_PREFETCH_MULTIPLIER</span> <span class="o">=</span> <span class="mi">1</span>
+ <span class="n">CELERY_ACKS_LATE</span> <span class="o">=</span> <span class="bp">True</span>
+ <span class="n">BROKER_URL</span> <span class="o">=</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'celery'</span><span class="p">,</span> <span class="s1">'BROKER_URL'</span><span class="p">)</span>
+ <span class="n">CELERY_RESULT_BACKEND</span> <span class="o">=</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'celery'</span><span class="p">,</span> <span class="s1">'CELERY_RESULT_BACKEND'</span><span class="p">)</span>
+ <span class="n">CELERYD_CONCURRENCY</span> <span class="o">=</span> <span class="n">configuration</span><span class="o">.</span><span class="n">getint</span><span class="p">(</span><span class="s1">'celery'</span><span class="p">,</span> <span class="s1">'CELERYD_CONCURRENCY'</span><span class="p">)</span>
+ <span class="n">CELERY_DEFAULT_QUEUE</span> <span class="o">=</span> <span class="n">DEFAULT_QUEUE</span>
+ <span class="n">CELERY_DEFAULT_EXCHANGE</span> <span class="o">=</span> <span class="n">DEFAULT_QUEUE</span>
+
+<span class="n">app</span> <span class="o">=</span> <span class="n">Celery</span><span class="p">(</span>
+ <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'celery'</span><span class="p">,</span> <span class="s1">'CELERY_APP_NAME'</span><span class="p">),</span>
+ <span class="n">config_source</span><span class="o">=</span><span class="n">CeleryConfig</span><span class="p">)</span>
+
+
+<span class="nd">@app.task</span>
+<span class="k">def</span> <span class="nf">execute_command</span><span class="p">(</span><span class="n">command</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Executing command in Celery "</span> <span class="o">+</span> <span class="n">command</span><span class="p">)</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">subprocess</span><span class="o">.</span><span class="n">check_call</span><span class="p">(</span><span class="n">command</span><span class="p">,</span> <span class="n">shell</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
+ <span class="k">except</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">CalledProcessError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Celery command failed'</span><span class="p">)</span>
+
+
+<div class="viewcode-block" id="CeleryExecutor"><a class="viewcode-back" href="../../../code.html#airflow.executors.CeleryExecutor">[docs]</a><span class="k">class</span> <span class="nc">CeleryExecutor</span><span class="p">(</span><span class="n">BaseExecutor</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> CeleryExecutor is recommended for production use of Airflow. It allows</span>
+<span class="sd"> distributing the execution of task instances to multiple worker nodes.</span>
+
+<span class="sd"> Celery is a simple, flexible and reliable distributed system to process</span>
+<span class="sd"> vast amounts of messages, while providing operations with the tools</span>
+<span class="sd"> required to maintain such a system.</span>
+<span class="sd"> """</span>
+
+ <span class="k">def</span> <span class="nf">start</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">tasks</span> <span class="o">=</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">last_state</span> <span class="o">=</span> <span class="p">{}</span>
+
+ <span class="k">def</span> <span class="nf">execute_async</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">command</span><span class="p">,</span> <span class="n">queue</span><span class="o">=</span><span class="n">DEFAULT_QUEUE</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span> <span class="s2">"[celery] queuing {key} through celery, "</span>
+ <span class="s2">"queue={queue}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">tasks</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">execute_command</span><span class="o">.</span><span class="n">apply_async</span><span class="p">(</span>
+ <span class="n">args</span><span class="o">=</span><span class="p">[</span><span class="n">command</span><span class="p">],</span> <span class="n">queue</span><span class="o">=</span><span class="n">queue</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">last_state</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">celery_states</span><span class="o">.</span><span class="n">PENDING</span>
+
+ <span class="k">def</span> <span class="nf">sync</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span>
+ <span class="s2">"Inquiring about {} celery task(s)"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">tasks</span><span class="p">)))</span>
+ <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">async</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">tasks</span><span class="o">.</span><span class="n">items</span><span class="p">()):</span>
+ <span class="n">state</span> <span class="o">=</span> <span class="n">async</span><span class="o">.</span><span class="n">state</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">last_state</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">!=</span> <span class="n">state</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">state</span> <span class="o">==</span> <span class="n">celery_states</span><span class="o">.</span><span class="n">SUCCESS</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">success</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
+ <span class="k">del</span> <span class="bp">self</span><span class="o">.</span><span class="n">tasks</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
+ <span class="k">del</span> <span class="bp">self</span><span class="o">.</span><span class="n">last_state</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
+ <span class="k">elif</span> <span class="n">state</span> <span class="o">==</span> <span class="n">celery_states</span><span class="o">.</span><span class="n">FAILURE</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">fail</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
+ <span class="k">del</span> <span class="bp">self</span><span class="o">.</span><span class="n">tasks</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
+ <span class="k">del</span> <span class="bp">self</span><span class="o">.</span><span class="n">last_state</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
+ <span class="k">elif</span> <span class="n">state</span> <span class="o">==</span> <span class="n">celery_states</span><span class="o">.</span><span class="n">REVOKED</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">fail</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
+ <span class="k">del</span> <span class="bp">self</span><span class="o">.</span><span class="n">tasks</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
+ <span class="k">del</span> <span class="bp">self</span><span class="o">.</span><span class="n">last_state</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Unexpected state: "</span> <span class="o">+</span> <span class="n">async</span><span class="o">.</span><span class="n">state</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">last_state</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">async</span><span class="o">.</span><span class="n">state</span>
+
+ <span class="k">def</span> <span class="nf">end</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">synchronous</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
+ <span class="k">if</span> <span class="n">synchronous</span><span class="p">:</span>
+ <span class="k">while</span> <span class="nb">any</span><span class="p">([</span>
+ <span class="n">async</span><span class="o">.</span><span class="n">state</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">celery_states</span><span class="o">.</span><span class="n">READY_STATES</span>
+ <span class="k">for</span> <span class="n">async</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">tasks</span><span class="o">.</span><span class="n">values</span><span class="p">()]):</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">5</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sync</span><span class="p">()</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/airflow/executors/local_executor.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/executors/local_executor.html b/_modules/airflow/executors/local_executor.html
new file mode 100644
index 0000000..efd28ef
--- /dev/null
+++ b/_modules/airflow/executors/local_executor.html
@@ -0,0 +1,276 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.executors.local_executor — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../../../index.html"/>
+ <link rel="up" title="Module code" href="../../index.html"/>
+
+
+ <script src="../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../index.html">Module code</a> »</li>
+
+ <li>airflow.executors.local_executor</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.executors.local_executor</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">multiprocessing</span>
+<span class="kn">import</span> <span class="nn">subprocess</span>
+<span class="kn">import</span> <span class="nn">time</span>
+
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">range</span>
+
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="kn">import</span> <span class="n">configuration</span>
+<span class="kn">from</span> <span class="nn">airflow.executors.base_executor</span> <span class="kn">import</span> <span class="n">BaseExecutor</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.state</span> <span class="kn">import</span> <span class="n">State</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.logging</span> <span class="kn">import</span> <span class="n">LoggingMixin</span>
+
+<span class="n">PARALLELISM</span> <span class="o">=</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'PARALLELISM'</span><span class="p">)</span>
+
+
+<span class="k">class</span> <span class="nc">LocalWorker</span><span class="p">(</span><span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="p">,</span> <span class="n">LoggingMixin</span><span class="p">):</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">task_queue</span><span class="p">,</span> <span class="n">result_queue</span><span class="p">):</span>
+ <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Process</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">task_queue</span> <span class="o">=</span> <span class="n">task_queue</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">result_queue</span> <span class="o">=</span> <span class="n">result_queue</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">daemon</span> <span class="o">=</span> <span class="bp">True</span>
+
+ <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
+ <span class="n">key</span><span class="p">,</span> <span class="n">command</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">task_queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">key</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
+ <span class="c1"># Received poison pill, no more tasks to run</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">task_queue</span><span class="o">.</span><span class="n">task_done</span><span class="p">()</span>
+ <span class="k">break</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"{} running {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">__class__</span><span class="o">.</span><span class="n">__name__</span><span class="p">,</span> <span class="n">command</span><span class="p">))</span>
+ <span class="n">command</span> <span class="o">=</span> <span class="s2">"exec bash -c '{0}'"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">command</span><span class="p">)</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">subprocess</span><span class="o">.</span><span class="n">check_call</span><span class="p">(</span><span class="n">command</span><span class="p">,</span> <span class="n">shell</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
+ <span class="n">state</span> <span class="o">=</span> <span class="n">State</span><span class="o">.</span><span class="n">SUCCESS</span>
+ <span class="k">except</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">CalledProcessError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="n">state</span> <span class="o">=</span> <span class="n">State</span><span class="o">.</span><span class="n">FAILED</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"failed to execute task {}:"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)))</span>
+ <span class="c1"># raise e</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">result_queue</span><span class="o">.</span><span class="n">put</span><span class="p">((</span><span class="n">key</span><span class="p">,</span> <span class="n">state</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">task_queue</span><span class="o">.</span><span class="n">task_done</span><span class="p">()</span>
+ <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
+
+
+<div class="viewcode-block" id="LocalExecutor"><a class="viewcode-back" href="../../../code.html#airflow.executors.LocalExecutor">[docs]</a><span class="k">class</span> <span class="nc">LocalExecutor</span><span class="p">(</span><span class="n">BaseExecutor</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> LocalExecutor executes tasks locally in parallel. It uses the</span>
+<span class="sd"> multiprocessing Python library and queues to parallelize the execution</span>
+<span class="sd"> of tasks.</span>
+<span class="sd"> """</span>
+
+ <span class="k">def</span> <span class="nf">start</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">queue</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">JoinableQueue</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">result_queue</span> <span class="o">=</span> <span class="n">multiprocessing</span><span class="o">.</span><span class="n">Queue</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">workers</span> <span class="o">=</span> <span class="p">[</span>
+ <span class="n">LocalWorker</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">queue</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">result_queue</span><span class="p">)</span>
+ <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">parallelism</span><span class="p">)</span>
+ <span class="p">]</span>
+
+ <span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">workers</span><span class="p">:</span>
+ <span class="n">w</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
+
+ <span class="k">def</span> <span class="nf">execute_async</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">command</span><span class="p">,</span> <span class="n">queue</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">queue</span><span class="o">.</span><span class="n">put</span><span class="p">((</span><span class="n">key</span><span class="p">,</span> <span class="n">command</span><span class="p">))</span>
+
+ <span class="k">def</span> <span class="nf">sync</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">while</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">result_queue</span><span class="o">.</span><span class="n">empty</span><span class="p">():</span>
+ <span class="n">results</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">result_queue</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">change_state</span><span class="p">(</span><span class="o">*</span><span class="n">results</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">end</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="c1"># Sending poison pill to all worker</span>
+ <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">queue</span><span class="o">.</span><span class="n">put</span><span class="p">((</span><span class="bp">None</span><span class="p">,</span> <span class="bp">None</span><span class="p">))</span> <span class="k">for</span> <span class="n">w</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">workers</span><span class="p">]</span>
+ <span class="c1"># Wait for commands to finish</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">queue</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sync</span><span class="p">()</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/airflow/executors/sequential_executor.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/executors/sequential_executor.html b/_modules/airflow/executors/sequential_executor.html
new file mode 100644
index 0000000..2d90305
--- /dev/null
+++ b/_modules/airflow/executors/sequential_executor.html
@@ -0,0 +1,238 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.executors.sequential_executor — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../../../index.html"/>
+ <link rel="up" title="Module code" href="../../index.html"/>
+
+
+ <script src="../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../index.html">Module code</a> »</li>
+
+ <li>airflow.executors.sequential_executor</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.executors.sequential_executor</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">str</span>
+<span class="kn">import</span> <span class="nn">subprocess</span>
+
+<span class="kn">from</span> <span class="nn">airflow.executors.base_executor</span> <span class="kn">import</span> <span class="n">BaseExecutor</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.state</span> <span class="kn">import</span> <span class="n">State</span>
+
+
+<div class="viewcode-block" id="SequentialExecutor"><a class="viewcode-back" href="../../../code.html#airflow.executors.SequentialExecutor">[docs]</a><span class="k">class</span> <span class="nc">SequentialExecutor</span><span class="p">(</span><span class="n">BaseExecutor</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> This executor will only run one task instance at a time, can be used</span>
+<span class="sd"> for debugging. It is also the only executor that can be used with sqlite</span>
+<span class="sd"> since sqlite doesn't support multiple connections.</span>
+
+<span class="sd"> Since we want airflow to work out of the box, it defaults to this</span>
+<span class="sd"> SequentialExecutor alongside sqlite as you first install it.</span>
+<span class="sd"> """</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SequentialExecutor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">()</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">commands_to_run</span> <span class="o">=</span> <span class="p">[]</span>
+
+ <span class="k">def</span> <span class="nf">execute_async</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">command</span><span class="p">,</span> <span class="n">queue</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">commands_to_run</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="n">key</span><span class="p">,</span> <span class="n">command</span><span class="p">,))</span>
+
+ <span class="k">def</span> <span class="nf">sync</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">command</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">commands_to_run</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Executing command: {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">command</span><span class="p">))</span>
+
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">subprocess</span><span class="o">.</span><span class="n">check_call</span><span class="p">(</span><span class="n">command</span><span class="p">,</span> <span class="n">shell</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">change_state</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">State</span><span class="o">.</span><span class="n">SUCCESS</span><span class="p">)</span>
+ <span class="k">except</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">CalledProcessError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">change_state</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">State</span><span class="o">.</span><span class="n">FAILED</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s2">"Failed to execute task {}:"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)))</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">commands_to_run</span> <span class="o">=</span> <span class="p">[]</span>
+
+ <span class="k">def</span> <span class="nf">end</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">heartbeat</span><span class="p">()</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/airflow/macros.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/macros.html b/_modules/airflow/macros.html
new file mode 100644
index 0000000..be420df
--- /dev/null
+++ b/_modules/airflow/macros.html
@@ -0,0 +1,255 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.macros — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../../index.html"/>
+ <link rel="up" title="Module code" href="../index.html"/>
+
+
+ <script src="../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../../index.html">Docs</a> »</li>
+
+ <li><a href="../index.html">Module code</a> »</li>
+
+ <li>airflow.macros</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.macros</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">absolute_import</span>
+<span class="kn">from</span> <span class="nn">random</span> <span class="kn">import</span> <span class="n">random</span>
+<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timedelta</span>
+<span class="kn">import</span> <span class="nn">dateutil</span>
+<span class="kn">import</span> <span class="nn">time</span>
+<span class="kn">from</span> <span class="nn">.</span> <span class="kn">import</span> <span class="n">hive</span>
+<span class="kn">import</span> <span class="nn">uuid</span>
+
+
+<div class="viewcode-block" id="ds_add"><a class="viewcode-back" href="../../code.html#airflow.macros.ds_add">[docs]</a><span class="k">def</span> <span class="nf">ds_add</span><span class="p">(</span><span class="n">ds</span><span class="p">,</span> <span class="n">days</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Add or subtract days from a YYYY-MM-DD</span>
+
+<span class="sd"> :param ds: anchor date in ``YYYY-MM-DD`` format to add to</span>
+<span class="sd"> :type ds: str</span>
+<span class="sd"> :param days: number of days to add to the ds, you can use negative values</span>
+<span class="sd"> :type days: int</span>
+
+<span class="sd"> >>> ds_add('2015-01-01', 5)</span>
+<span class="sd"> '2015-01-06'</span>
+<span class="sd"> >>> ds_add('2015-01-06', -5)</span>
+<span class="sd"> '2015-01-01'</span>
+<span class="sd"> """</span>
+
+ <span class="n">ds</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">ds</span><span class="p">,</span> <span class="s1">'%Y-%m-</span><span class="si">%d</span><span class="s1">'</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">days</span><span class="p">:</span>
+ <span class="n">ds</span> <span class="o">=</span> <span class="n">ds</span> <span class="o">+</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">ds</span><span class="o">.</span><span class="n">isoformat</span><span class="p">()[:</span><span class="mi">10</span><span class="p">]</span></div>
+
+
+<div class="viewcode-block" id="ds_format"><a class="viewcode-back" href="../../code.html#airflow.macros.ds_format">[docs]</a><span class="k">def</span> <span class="nf">ds_format</span><span class="p">(</span><span class="n">ds</span><span class="p">,</span> <span class="n">input_format</span><span class="p">,</span> <span class="n">output_format</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Takes an input string and outputs another string</span>
+<span class="sd"> as specified in the output format</span>
+
+<span class="sd"> :param ds: input string which contains a date</span>
+<span class="sd"> :type ds: str</span>
+<span class="sd"> :param input_format: input string format. E.g. %Y-%m-%d</span>
+<span class="sd"> :type input_format: str</span>
+<span class="sd"> :param output_format: output string format E.g. %Y-%m-%d</span>
+<span class="sd"> :type output_format: str</span>
+
+<span class="sd"> >>> ds_format('2015-01-01', "%Y-%m-%d", "%m-%d-%y")</span>
+<span class="sd"> '01-01-15'</span>
+<span class="sd"> >>> ds_format('1/5/2015', "%m/%d/%Y", "%Y-%m-%d")</span>
+<span class="sd"> '2015-01-05'</span>
+<span class="sd"> """</span>
+ <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">ds</span><span class="p">,</span> <span class="n">input_format</span><span class="p">)</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="n">output_format</span><span class="p">)</span></div>
+
+
+<div class="viewcode-block" id="integrate_plugins"><a class="viewcode-back" href="../../code.html#airflow.macros.integrate_plugins">[docs]</a><span class="k">def</span> <span class="nf">integrate_plugins</span><span class="p">():</span>
+ <span class="sd">"""Integrate plugins to the context"""</span>
+ <span class="kn">from</span> <span class="nn">airflow.plugins_manager</span> <span class="kn">import</span> <span class="n">macros</span> <span class="k">as</span> <span class="n">_macros</span>
+ <span class="k">for</span> <span class="n">_macro</span> <span class="ow">in</span> <span class="n">_macros</span><span class="p">:</span>
+ <span class="nb">globals</span><span class="p">()[</span><span class="n">_macro</span><span class="o">.</span><span class="n">__name__</span><span class="p">]</span> <span class="o">=</span> <span class="n">_macro</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[07/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/cli.html
----------------------------------------------------------------------
diff --git a/cli.html b/cli.html
new file mode 100644
index 0000000..d604510
--- /dev/null
+++ b/cli.html
@@ -0,0 +1,1035 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Command Line Interface — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="Scheduling & Triggers" href="scheduler.html"/>
+ <link rel="prev" title="Data Profiling" href="profiling.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>Command Line Interface</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/cli.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="command-line-interface">
+<h1>Command Line Interface<a class="headerlink" href="#command-line-interface" title="Permalink to this headline">�</a></h1>
+<p>Airflow has a very rich command line interface that allows for
+many types of operation on a DAG, starting services, and supporting
+development and testing.</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span>
+ <span class="p">{</span><span class="n">resetdb</span><span class="p">,</span><span class="n">render</span><span class="p">,</span><span class="n">variables</span><span class="p">,</span><span class="n">pause</span><span class="p">,</span><span class="n">version</span><span class="p">,</span><span class="n">initdb</span><span class="p">,</span><span class="n">test</span><span class="p">,</span><span class="n">unpause</span><span class="p">,</span><span class="n">run</span><span class="p">,</span><span class="n">list_tasks</span><span class="p">,</span><span class="n">backfill</span><span class="p">,</span><span class="n">list_dags</span><span class="p">,</span><span class="n">kerberos</span><span class="p">,</span><span class="n">worker</span><span class="p">,</span><span class="n">webserver</span><span class="p">,</span><span class="n">flower</span><span class="p">,</span><span class="n">scheduler</span><span class="p">,</span><span class="n">task_state</span><span c
lass="p">,</span><span class="n">trigger_dag</span><span class="p">,</span><span class="n">serve_logs</span><span class="p">,</span><span class="n">clear</span><span class="p">,</span><span class="n">upgradedb</span><span class="p">}</span>
+ <span class="o">...</span>
+</pre></div>
+</div>
+<dl class="docutils">
+<dt>Sub-commands:</dt>
+<dd><dl class="first last docutils">
+<dt><strong>resetdb</strong></dt>
+<dd><p class="first">Burn down and rebuild the metadata database</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">resetdb</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">y</span><span class="p">]</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-y=False</span>, <span class="option">--yes=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Do not prompt to confirm reset. Use with care!</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>render</strong></dt>
+<dd><p class="first">Render a task instance’s template(s)</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">render</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span> <span class="n">task_id</span> <span class="n">execution_date</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Positional arguments:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group">
+<kbd>dag_id</kbd></td>
+<td>The id of the dag</td></tr>
+<tr><td class="option-group">
+<kbd>task_id</kbd></td>
+<td>The id of the task</td></tr>
+<tr><td class="option-group">
+<kbd>execution_date</kbd></td>
+<td>The execution date of the DAG</td></tr>
+</tbody>
+</table>
+</dd>
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
+</tr>
+<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>variables</strong></dt>
+<dd><p class="first">List all variables</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">variables</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">s</span> <span class="n">KEY</span> <span class="n">VAL</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">g</span> <span class="n">KEY</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">j</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">d</span> <span class="n">VAL</span><span class="p">]</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group">
+<kbd><span class="option">-s</span>, <span class="option">--set</span></kbd></td>
+<td>Set a variable</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">-g</span>, <span class="option">--get</span></kbd></td>
+<td>Get value of a variable</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-j=False</span>, <span class="option">--json=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Deserialize JSON variable</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">-d</span>, <span class="option">--default</span></kbd></td>
+<td>Default value returned if variable does not exist</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>pause</strong></dt>
+<dd><p class="first">Pause a DAG</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">pause</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Positional arguments:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group">
+<kbd>dag_id</kbd></td>
+<td>The id of the dag</td></tr>
+</tbody>
+</table>
+</dd>
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
+</tr>
+<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>version</strong></dt>
+<dd><p class="first">Show the version</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">version</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+</dl>
+</dd>
+<dt><strong>initdb</strong></dt>
+<dd><p class="first">Initialize the metadata database</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">initdb</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+</dl>
+</dd>
+<dt><strong>test</strong></dt>
+<dd><p class="first">Test a task instance. This will run a task without checking for dependencies or recording it’s state in the database.</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">test</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">dr</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">tp</span> <span class="n">TASK_PARAMS</span><span class="p">]</span>
+ <span class="n">dag_id</span> <span class="n">task_id</span> <span class="n">execution_date</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Positional arguments:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group">
+<kbd>dag_id</kbd></td>
+<td>The id of the dag</td></tr>
+<tr><td class="option-group">
+<kbd>task_id</kbd></td>
+<td>The id of the task</td></tr>
+<tr><td class="option-group">
+<kbd>execution_date</kbd></td>
+<td>The execution date of the DAG</td></tr>
+</tbody>
+</table>
+</dd>
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
+</tr>
+<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-dr=False</span>, <span class="option">--dry_run=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Perform a dry run</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-tp</span>, <span class="option">--task_params</span></kbd></td>
+</tr>
+<tr><td> </td><td>Sends a JSON params dict to the task</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>unpause</strong></dt>
+<dd><p class="first">Pause a DAG</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">unpause</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Positional arguments:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group">
+<kbd>dag_id</kbd></td>
+<td>The id of the dag</td></tr>
+</tbody>
+</table>
+</dd>
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
+</tr>
+<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>run</strong></dt>
+<dd><p class="first">Run a single task instance</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">run</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">m</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">f</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">pool</span> <span class="n">POOL</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">i</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">I</span>
<span class="p">]</span>
+ <span class="p">[</span><span class="o">--</span><span class="n">ship_dag</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">p</span> <span class="n">PICKLE</span><span class="p">]</span>
+ <span class="n">dag_id</span> <span class="n">task_id</span> <span class="n">execution_date</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Positional arguments:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group">
+<kbd>dag_id</kbd></td>
+<td>The id of the dag</td></tr>
+<tr><td class="option-group">
+<kbd>task_id</kbd></td>
+<td>The id of the task</td></tr>
+<tr><td class="option-group">
+<kbd>execution_date</kbd></td>
+<td>The execution date of the DAG</td></tr>
+</tbody>
+</table>
+</dd>
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
+</tr>
+<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-m=False</span>, <span class="option">--mark_success=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Mark jobs as succeeded without running them</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-f=False</span>, <span class="option">--force=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Force a run regardless or previous success</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--pool</span></kbd></td>
+<td>Resource pool to use</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-l=False</span>, <span class="option">--local=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Run the task using the LocalExecutor</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-i=False</span>, <span class="option">--ignore_dependencies=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Ignore upstream and depends_on_past dependencies</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-I=False</span>, <span class="option">--ignore_depends_on_past=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Ignore depends_on_past dependencies (but respect upstream dependencies)</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">--ship_dag=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Pickles (serializes) the DAG and ships it to the worker</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">-p</span>, <span class="option">--pickle</span></kbd></td>
+<td>Serialized pickle object of the entire dag (used internally)</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>list_tasks</strong></dt>
+<dd><p class="first">List the tasks within a DAG</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">list_tasks</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">t</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Positional arguments:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group">
+<kbd>dag_id</kbd></td>
+<td>The id of the dag</td></tr>
+</tbody>
+</table>
+</dd>
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-t=False</span>, <span class="option">--tree=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Tree view</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
+</tr>
+<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>backfill</strong></dt>
+<dd><p class="first">Run subsections of a DAG for a specified date range</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">backfill</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">t</span> <span class="n">TASK_REGEX</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">s</span> <span class="n">START_DATE</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">e</span> <span class="n">END_DATE</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">-</span><span class="n">m</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">x</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">a</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">i</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">I</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">--</span><span class="n">pool</span> <span class="n">POOL</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">dr</span><span class="p">]</span>
+ <span class="n">dag_id</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Positional arguments:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group">
+<kbd>dag_id</kbd></td>
+<td>The id of the dag</td></tr>
+</tbody>
+</table>
+</dd>
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-t</span>, <span class="option">--task_regex</span></kbd></td>
+</tr>
+<tr><td> </td><td>The regex to filter specific task_ids to backfill (optional)</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-s</span>, <span class="option">--start_date</span></kbd></td>
+</tr>
+<tr><td> </td><td>Override start_date YYYY-MM-DD</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">-e</span>, <span class="option">--end_date</span></kbd></td>
+<td>Override end_date YYYY-MM-DD</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-m=False</span>, <span class="option">--mark_success=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Mark jobs as succeeded without running them</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-l=False</span>, <span class="option">--local=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Run the task using the LocalExecutor</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-x=False</span>, <span class="option">--donot_pickle=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Do not attempt to pickle the DAG object to send over to the workers, just tell the workers to run their version of the code.</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-a=False</span>, <span class="option">--include_adhoc=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Include dags with the adhoc parameter.</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-i=False</span>, <span class="option">--ignore_dependencies=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Skip upstream tasks, run only the tasks matching the regexp. Only works in conjunction with task_regex</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-I=False</span>, <span class="option">--ignore_first_depends_on_past=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Ignores depends_on_past dependencies for the first set of tasks only (subsequent executions in the backfill DO respect depends_on_past).</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
+</tr>
+<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--pool</span></kbd></td>
+<td>Resource pool to use</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-dr=False</span>, <span class="option">--dry_run=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Perform a dry run</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>list_dags</strong></dt>
+<dd><p class="first">List all the DAGs</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">list_dags</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
+</tr>
+<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>kerberos</strong></dt>
+<dd><p class="first">Start a kerberos ticket renewer</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">kerberos</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">kt</span> <span class="p">[</span><span class="n">KEYTAB</span><span class="p">]]</span> <span class="p">[</span><span class="o">--</span><span class="n">pid</span> <span class="p">[</span><span class="n">PID</span><span class="p">]]</span> <span class="p">[</span><span class="o">-</span><span class="n">D</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">--</span><span class="n">stdout</span> <span class="n">STDOUT</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">stderr</span> <span class="n">STDERR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span> <span class="n">LOG_FILE</span><span class="p">]</span>
+ <span class="p">[</span><span class="n">principal</span><span class="p">]</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Positional arguments:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group">
+<kbd>principal</kbd></td>
+<td>kerberos principal</td></tr>
+</tbody>
+</table>
+</dd>
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-kt=airflow.keytab</span>, <span class="option">--keytab=airflow.keytab</span></kbd></td>
+</tr>
+<tr><td> </td><td>keytab</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--pid</span></kbd></td>
+<td>PID file location</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-D=False</span>, <span class="option">--daemon=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Daemonize instead of running on the foreground</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--stdout</span></kbd></td>
+<td>Redirect stdout to this file</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--stderr</span></kbd></td>
+<td>Redirect stderr to this file</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">-l</span>, <span class="option">--log-file</span></kbd></td>
+<td>Location of the log file</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>worker</strong></dt>
+<dd><p class="first">Start a Celery worker node</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">worker</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">p</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">q</span> <span class="n">QUEUES</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">c</span> <span class="n">CONCURRENCY</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">pid</span> <span class="p">[</span><span class="n">PID</span><span class="p">]]</span>
+ <span class="p">[</span><span class="o">-</span><span class="n">D</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">stdout</span> <span class="n">STDOUT</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">stderr</span> <span class="n">STDERR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span> <span class="n">LOG_FILE</span><span class="p">]</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-p=False</span>, <span class="option">--do_pickle=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Attempt to pickle the DAG object to send over to the workers, instead of letting workers run their version of the code.</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-q=default</span>, <span class="option">--queues=default</span></kbd></td>
+</tr>
+<tr><td> </td><td>Comma delimited list of queues to serve</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-c=16</span>, <span class="option">--concurrency=16</span></kbd></td>
+</tr>
+<tr><td> </td><td>The number of worker processes</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--pid</span></kbd></td>
+<td>PID file location</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-D=False</span>, <span class="option">--daemon=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Daemonize instead of running on the foreground</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--stdout</span></kbd></td>
+<td>Redirect stdout to this file</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--stderr</span></kbd></td>
+<td>Redirect stderr to this file</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">-l</span>, <span class="option">--log-file</span></kbd></td>
+<td>Location of the log file</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>webserver</strong></dt>
+<dd><p class="first">Start a Airflow webserver instance</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">webserver</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">p</span> <span class="n">PORT</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">w</span> <span class="n">WORKERS</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">-</span><span class="n">k</span> <span class="p">{</span><span class="n">sync</span><span class="p">,</span><span class="n">eventlet</span><span class="p">,</span><span class="n">gevent</span><span class="p">,</span><span class="n">tornado</span><span class="p">}]</span>
+ <span class="p">[</span><span class="o">-</span><span class="n">t</span> <span class="n">WORKER_TIMEOUT</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">hn</span> <span class="n">HOSTNAME</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">pid</span> <span class="p">[</span><span class="n">PID</span><span class="p">]]</span> <span class="p">[</span><span class="o">-</span><span class="n">D</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">--</span><span class="n">stdout</span> <span class="n">STDOUT</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">stderr</span> <span class="n">STDERR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span> <span class="n">LOG_FILE</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">-</span><span class="n">d</span><span class="p">]</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-p=8080</span>, <span class="option">--port=8080</span></kbd></td>
+</tr>
+<tr><td> </td><td>The port on which to run the server</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-w=4</span>, <span class="option">--workers=4</span></kbd></td>
+</tr>
+<tr><td> </td><td>Number of workers to run the webserver on</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-k=sync</span>, <span class="option">--workerclass=sync</span></kbd></td>
+</tr>
+<tr><td> </td><td><p class="first">The worker class to use for gunicorn</p>
+<p class="last">Possible choices: sync, eventlet, gevent, tornado</p>
+</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-t=120</span>, <span class="option">--worker_timeout=120</span></kbd></td>
+</tr>
+<tr><td> </td><td>The timeout for waiting on webserver workers</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-hn=0.0.0.0</span>, <span class="option">--hostname=0.0.0.0</span></kbd></td>
+</tr>
+<tr><td> </td><td>Set the hostname on which to run the web server</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--pid</span></kbd></td>
+<td>PID file location</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-D=False</span>, <span class="option">--daemon=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Daemonize instead of running on the foreground</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--stdout</span></kbd></td>
+<td>Redirect stdout to this file</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--stderr</span></kbd></td>
+<td>Redirect stderr to this file</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">-l</span>, <span class="option">--log-file</span></kbd></td>
+<td>Location of the log file</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-d=False</span>, <span class="option">--debug=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Use the server that ships with Flask in debug mode</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>flower</strong></dt>
+<dd><p class="first">Start a Celery Flower</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">flower</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">p</span> <span class="n">PORT</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">a</span> <span class="n">BROKER_API</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">pid</span> <span class="p">[</span><span class="n">PID</span><span class="p">]]</span> <span class="p">[</span><span class="o">-</span><span class="n">D</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">--</span><span class="n">stdout</span> <span class="n">STDOUT</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">stderr</span> <span class="n">STDERR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span> <span class="n">LOG_FILE</span><span class="p">]</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-p=5555</span>, <span class="option">--port=5555</span></kbd></td>
+</tr>
+<tr><td> </td><td>The port on which to run the server</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-a</span>, <span class="option">--broker_api</span></kbd></td>
+</tr>
+<tr><td> </td><td>Broker api</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--pid</span></kbd></td>
+<td>PID file location</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-D=False</span>, <span class="option">--daemon=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Daemonize instead of running on the foreground</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--stdout</span></kbd></td>
+<td>Redirect stdout to this file</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--stderr</span></kbd></td>
+<td>Redirect stderr to this file</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">-l</span>, <span class="option">--log-file</span></kbd></td>
+<td>Location of the log file</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>scheduler</strong></dt>
+<dd><p class="first">Start a scheduler scheduler instance</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">scheduler</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">d</span> <span class="n">DAG_ID</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">n</span> <span class="n">NUM_RUNS</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">p</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">--</span><span class="n">pid</span> <span class="p">[</span><span class="n">PID</span><span class="p">]]</span> <span class="p">[</span><span class="o">-</span><span class="n">D</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">stdout</span> <span class="n">STDOUT</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">--</span><span class="n">stderr</span> <span class="n">STDERR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">l</span> <span class="n">LOG_FILE</span><span class="p">]</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group">
+<kbd><span class="option">-d</span>, <span class="option">--dag_id</span></kbd></td>
+<td>The id of the dag to run</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
+</tr>
+<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">-n</span>, <span class="option">--num_runs</span></kbd></td>
+<td>Set the number of runs to execute before exiting</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-p=False</span>, <span class="option">--do_pickle=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Attempt to pickle the DAG object to send over to the workers, instead of letting workers run their version of the code.</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--pid</span></kbd></td>
+<td>PID file location</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-D=False</span>, <span class="option">--daemon=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Daemonize instead of running on the foreground</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--stdout</span></kbd></td>
+<td>Redirect stdout to this file</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">--stderr</span></kbd></td>
+<td>Redirect stderr to this file</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">-l</span>, <span class="option">--log-file</span></kbd></td>
+<td>Location of the log file</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>task_state</strong></dt>
+<dd><p class="first">Get the status of a task instance</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">task_state</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="n">dag_id</span> <span class="n">task_id</span> <span class="n">execution_date</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Positional arguments:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group">
+<kbd>dag_id</kbd></td>
+<td>The id of the dag</td></tr>
+<tr><td class="option-group">
+<kbd>task_id</kbd></td>
+<td>The id of the task</td></tr>
+<tr><td class="option-group">
+<kbd>execution_date</kbd></td>
+<td>The execution date of the DAG</td></tr>
+</tbody>
+</table>
+</dd>
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
+</tr>
+<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>trigger_dag</strong></dt>
+<dd><p class="first">Trigger a DAG run</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">trigger_dag</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">r</span> <span class="n">RUN_ID</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">c</span> <span class="n">CONF</span><span class="p">]</span> <span class="n">dag_id</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Positional arguments:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group">
+<kbd>dag_id</kbd></td>
+<td>The id of the dag</td></tr>
+</tbody>
+</table>
+</dd>
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
+</tr>
+<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">-r</span>, <span class="option">--run_id</span></kbd></td>
+<td>Helps to indentify this run</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">-c</span>, <span class="option">--conf</span></kbd></td>
+<td>json string that gets pickled into the DagRun’s conf attribute</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>serve_logs</strong></dt>
+<dd><p class="first">Serve logs generate by worker</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">serve_logs</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+</dl>
+</dd>
+<dt><strong>clear</strong></dt>
+<dd><p class="first">Clear a set of task instance, as if they never ran</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">clear</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">t</span> <span class="n">TASK_REGEX</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">s</span> <span class="n">START_DATE</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">e</span> <span class="n">END_DATE</span><span class="p">]</span>
+ <span class="p">[</span><span class="o">-</span><span class="n">sd</span> <span class="n">SUBDIR</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">u</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">d</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">c</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">f</span><span class="p">]</span> <span class="p">[</span><span class="o">-</span><span class="n">r</span><span class="p">]</span>
+ <span class="n">dag_id</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+<dt>Positional arguments:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group">
+<kbd>dag_id</kbd></td>
+<td>The id of the dag</td></tr>
+</tbody>
+</table>
+</dd>
+<dt>Options:</dt>
+<dd><table class="first last docutils option-list" frame="void" rules="none">
+<col class="option" />
+<col class="description" />
+<tbody valign="top">
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-t</span>, <span class="option">--task_regex</span></kbd></td>
+</tr>
+<tr><td> </td><td>The regex to filter specific task_ids to backfill (optional)</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-s</span>, <span class="option">--start_date</span></kbd></td>
+</tr>
+<tr><td> </td><td>Override start_date YYYY-MM-DD</td></tr>
+<tr><td class="option-group">
+<kbd><span class="option">-e</span>, <span class="option">--end_date</span></kbd></td>
+<td>Override end_date YYYY-MM-DD</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-sd=/Users/maxime_beauchemin/dags</span>, <span class="option">--subdir=/Users/maxime_beauchemin/dags</span></kbd></td>
+</tr>
+<tr><td> </td><td>File location or directory from which to look for the dag</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-u=False</span>, <span class="option">--upstream=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Include upstream tasks</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-d=False</span>, <span class="option">--downstream=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Include downstream tasks</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-c=False</span>, <span class="option">--no_confirm=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Do not request confirmation</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-f=False</span>, <span class="option">--only_failed=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Only failed jobs</td></tr>
+<tr><td class="option-group" colspan="2">
+<kbd><span class="option">-r=False</span>, <span class="option">--only_running=False</span></kbd></td>
+</tr>
+<tr><td> </td><td>Only running jobs</td></tr>
+</tbody>
+</table>
+</dd>
+</dl>
+</dd>
+<dt><strong>upgradedb</strong></dt>
+<dd><p class="first">Upgrade metadata database to latest version</p>
+<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">usage</span><span class="p">:</span> <span class="n">airflow</span> <span class="n">upgradedb</span> <span class="p">[</span><span class="o">-</span><span class="n">h</span><span class="p">]</span>
+</pre></div>
+</div>
+<dl class="last docutils">
+</dl>
+</dd>
+</dl>
+</dd>
+</dl>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="scheduler.html" class="btn btn-neutral float-right" title="Scheduling & Triggers" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="profiling.html" class="btn btn-neutral" title="Data Profiling" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[03/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/installation.html
----------------------------------------------------------------------
diff --git a/installation.html b/installation.html
new file mode 100644
index 0000000..19aece3
--- /dev/null
+++ b/installation.html
@@ -0,0 +1,358 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Installation — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="Tutorial" href="tutorial.html"/>
+ <link rel="prev" title="Quick Start" href="start.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">Installation</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#getting-airflow">Getting Airflow</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#extra-packages">Extra Packages</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>Installation</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/installation.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="installation">
+<h1>Installation<a class="headerlink" href="#installation" title="Permalink to this headline">�</a></h1>
+<div class="section" id="getting-airflow">
+<h2>Getting Airflow<a class="headerlink" href="#getting-airflow" title="Permalink to this headline">�</a></h2>
+<p>The easiest way to install the latest stable version of Airflow is with <code class="docutils literal"><span class="pre">pip</span></code>:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span>pip install airflow
+</pre></div>
+</div>
+<p>You can also install Airflow with support for extra features like <code class="docutils literal"><span class="pre">s3</span></code> or <code class="docutils literal"><span class="pre">postgres</span></code>:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span>pip install <span class="s2">"airflow[s3, postgres]"</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="extra-packages">
+<h2>Extra Packages<a class="headerlink" href="#extra-packages" title="Permalink to this headline">�</a></h2>
+<p>The <code class="docutils literal"><span class="pre">airflow</span></code> PyPI basic package only installs what’s needed to get started.
+Subpackages can be installed depending on what will be useful in your
+environment. For instance, if you don’t need connectivity with Postgres,
+you won’t have to go through the trouble of installing the <code class="docutils literal"><span class="pre">postgres-devel</span></code>
+yum package, or whatever equivalent applies on the distribution you are using.</p>
+<p>Behind the scenes, Airflow does conditional imports of operators that require
+these extra dependencies.</p>
+<p>Here’s the list of the subpackages and what they enable:</p>
+<table border="1" class="docutils">
+<colgroup>
+<col width="15%" />
+<col width="37%" />
+<col width="49%" />
+</colgroup>
+<thead valign="bottom">
+<tr class="row-odd"><th class="head">subpackage</th>
+<th class="head">install command</th>
+<th class="head">enables</th>
+</tr>
+</thead>
+<tbody valign="top">
+<tr class="row-even"><td>all</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[all]</span></code></td>
+<td>All Airflow features known to man</td>
+</tr>
+<tr class="row-odd"><td>all_dbs</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[all_dbs]</span></code></td>
+<td>All databases integrations</td>
+</tr>
+<tr class="row-even"><td>async</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[async]</span></code></td>
+<td>Async worker classes for gunicorn</td>
+</tr>
+<tr class="row-odd"><td>devel</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[devel]</span></code></td>
+<td>Minimum dev tools requirements</td>
+</tr>
+<tr class="row-even"><td>devel_hadoop</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[devel_hadoop]</span></code></td>
+<td>Airflow + dependencies on the Hadoop stack</td>
+</tr>
+<tr class="row-odd"><td>celery</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[celery]</span></code></td>
+<td>CeleryExecutor</td>
+</tr>
+<tr class="row-even"><td>crypto</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[crypto]</span></code></td>
+<td>Encrypt connection passwords in metadata db</td>
+</tr>
+<tr class="row-odd"><td>druid</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[druid]</span></code></td>
+<td>Druid.io related operators & hooks</td>
+</tr>
+<tr class="row-even"><td>gcp_api</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[gcp_api]</span></code></td>
+<td>Google Cloud Platform hooks and operators
+(using <code class="docutils literal"><span class="pre">google-api-python-client</span></code>)</td>
+</tr>
+<tr class="row-odd"><td>jdbc</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[jdbc]</span></code></td>
+<td>JDBC hooks and operators</td>
+</tr>
+<tr class="row-even"><td>hdfs</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[hdfs]</span></code></td>
+<td>HDFS hooks and operators</td>
+</tr>
+<tr class="row-odd"><td>hive</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[hive]</span></code></td>
+<td>All Hive related operators</td>
+</tr>
+<tr class="row-even"><td>kerberos</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[kerberos]</span></code></td>
+<td>kerberos integration for kerberized hadoop</td>
+</tr>
+<tr class="row-odd"><td>ldap</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[ldap]</span></code></td>
+<td>ldap authentication for users</td>
+</tr>
+<tr class="row-even"><td>mssql</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[mssql]</span></code></td>
+<td>Microsoft SQL operators and hook,
+support as an Airflow backend</td>
+</tr>
+<tr class="row-odd"><td>mysql</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[mysql]</span></code></td>
+<td>MySQL operators and hook, support as
+an Airflow backend</td>
+</tr>
+<tr class="row-even"><td>password</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[password]</span></code></td>
+<td>Password Authentication for users</td>
+</tr>
+<tr class="row-odd"><td>postgres</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[postgres]</span></code></td>
+<td>Postgres operators and hook, support
+as an Airflow backend</td>
+</tr>
+<tr class="row-even"><td>qds</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[qds]</span></code></td>
+<td>Enable QDS (qubole data services) support</td>
+</tr>
+<tr class="row-odd"><td>rabbitmq</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[rabbitmq]</span></code></td>
+<td>Rabbitmq support as a Celery backend</td>
+</tr>
+<tr class="row-even"><td>s3</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[s3]</span></code></td>
+<td><code class="docutils literal"><span class="pre">S3KeySensor</span></code>, <code class="docutils literal"><span class="pre">S3PrefixSensor</span></code></td>
+</tr>
+<tr class="row-odd"><td>samba</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[samba]</span></code></td>
+<td><code class="docutils literal"><span class="pre">Hive2SambaOperator</span></code></td>
+</tr>
+<tr class="row-even"><td>slack</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[slack]</span></code></td>
+<td><code class="docutils literal"><span class="pre">SlackAPIPostOperator</span></code></td>
+</tr>
+<tr class="row-odd"><td>vertica</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[vertica]</span></code></td>
+<td>Vertica hook
+support as an Airflow backend</td>
+</tr>
+<tr class="row-even"><td>cloudant</td>
+<td><code class="docutils literal"><span class="pre">pip</span> <span class="pre">install</span> <span class="pre">airflow[cloudant]</span></code></td>
+<td>Cloudant hook</td>
+</tr>
+</tbody>
+</table>
+</div>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="tutorial.html" class="btn btn-neutral float-right" title="Tutorial" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="start.html" class="btn btn-neutral" title="Quick Start" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/license.html
----------------------------------------------------------------------
diff --git a/license.html b/license.html
new file mode 100644
index 0000000..deec925
--- /dev/null
+++ b/license.html
@@ -0,0 +1,418 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>License — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="Quick Start" href="start.html"/>
+ <link rel="prev" title="Project" href="project.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>License</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/license.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="license">
+<h1>License<a class="headerlink" href="#license" title="Permalink to this headline">�</a></h1>
+<a class="reference internal image-reference" href="_images/apache.jpg"><img alt="_images/apache.jpg" src="_images/apache.jpg" style="width: 150px;" /></a>
+<div class="highlight-default"><div class="highlight"><pre><span></span>Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+Copyright 2015 Apache Software Foundation
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+Status API Training Shop Blog About
+� 2016 GitHub, Inc. Terms Privacy Security Contact Help
+</pre></div>
+</div>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="start.html" class="btn btn-neutral float-right" title="Quick Start" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="project.html" class="btn btn-neutral" title="Project" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/objects.inv
----------------------------------------------------------------------
diff --git a/objects.inv b/objects.inv
new file mode 100644
index 0000000..ea45f19
Binary files /dev/null and b/objects.inv differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/plugins.html
----------------------------------------------------------------------
diff --git a/plugins.html b/plugins.html
new file mode 100644
index 0000000..854b245
--- /dev/null
+++ b/plugins.html
@@ -0,0 +1,343 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Plugins — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="Security" href="security.html"/>
+ <link rel="prev" title="Scheduling & Triggers" href="scheduler.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">Plugins</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#what-for">What for?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#why-build-on-top-of-airflow">Why build on top of Airflow?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#interface">Interface</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#example">Example</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>Plugins</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/plugins.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="plugins">
+<h1>Plugins<a class="headerlink" href="#plugins" title="Permalink to this headline">�</a></h1>
+<p>Airflow has a simple plugin manager built-in that can integrate external
+features to its core by simply dropping files in your
+<code class="docutils literal"><span class="pre">$AIRFLOW_HOME/plugins</span></code> folder.</p>
+<p>The python modules in the <code class="docutils literal"><span class="pre">plugins</span></code> folder get imported,
+and <strong>hooks</strong>, <strong>operators</strong>, <strong>macros</strong>, <strong>executors</strong> and web <strong>views</strong>
+get integrated to Airflow’s main collections and become available for use.</p>
+<div class="section" id="what-for">
+<h2>What for?<a class="headerlink" href="#what-for" title="Permalink to this headline">�</a></h2>
+<p>Airflow offers a generic toolbox for working with data. Different
+organizations have different stacks and different needs. Using Airflow
+plugins can be a way for companies to customize their Airflow installation
+to reflect their ecosystem.</p>
+<p>Plugins can be used as an easy way to write, share and activate new sets of
+features.</p>
+<p>There’s also a need for a set of more complex applications to interact with
+different flavors of data and metadata.</p>
+<p>Examples:</p>
+<ul class="simple">
+<li>A set of tools to parse Hive logs and expose Hive metadata (CPU /IO / phases/ skew /...)</li>
+<li>An anomaly detection framework, allowing people to collect metrics, set thresholds and alerts</li>
+<li>An auditing tool, helping understand who accesses what</li>
+<li>A config-driven SLA monitoring tool, allowing you to set monitored tables and at what time
+they should land, alert people, and expose visualizations of outages</li>
+<li>...</li>
+</ul>
+</div>
+<div class="section" id="why-build-on-top-of-airflow">
+<h2>Why build on top of Airflow?<a class="headerlink" href="#why-build-on-top-of-airflow" title="Permalink to this headline">�</a></h2>
+<p>Airflow has many components that can be reused when building an application:</p>
+<ul class="simple">
+<li>A web server you can use to render your views</li>
+<li>A metadata database to store your models</li>
+<li>Access to your databases, and knowledge of how to connect to them</li>
+<li>An array of workers that your application can push workload to</li>
+<li>Airflow is deployed, you can just piggy back on it’s deployment logistics</li>
+<li>Basic charting capabilities, underlying libraries and abstractions</li>
+</ul>
+</div>
+<div class="section" id="interface">
+<h2>Interface<a class="headerlink" href="#interface" title="Permalink to this headline">�</a></h2>
+<p>To create a plugin you will need to derive the
+<code class="docutils literal"><span class="pre">airflow.plugins_manager.AirflowPlugin</span></code> class and reference the objects
+you want to plug into Airflow. Here’s what the class you need to derive
+looks like:</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">AirflowPlugin</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
+ <span class="c1"># The name of your plugin (str)</span>
+ <span class="n">name</span> <span class="o">=</span> <span class="kc">None</span>
+ <span class="c1"># A list of class(es) derived from BaseOperator</span>
+ <span class="n">operators</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="c1"># A list of class(es) derived from BaseHook</span>
+ <span class="n">hooks</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="c1"># A list of class(es) derived from BaseExecutor</span>
+ <span class="n">executors</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="c1"># A list of references to inject into the macros namespace</span>
+ <span class="n">macros</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="c1"># A list of objects created from a class derived</span>
+ <span class="c1"># from flask_admin.BaseView</span>
+ <span class="n">admin_views</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="c1"># A list of Blueprint object created from flask.Blueprint</span>
+ <span class="n">flask_blueprints</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="c1"># A list of menu links (flask_admin.base.MenuLink)</span>
+ <span class="n">menu_links</span> <span class="o">=</span> <span class="p">[]</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="example">
+<h2>Example<a class="headerlink" href="#example" title="Permalink to this headline">�</a></h2>
+<p>The code below defines a plugin that injects a set of dummy object
+definitions in Airflow.</p>
+<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="c1"># This is the class you derive to create a plugin</span>
+<span class="kn">from</span> <span class="nn">airflow.plugins_manager</span> <span class="k">import</span> <span class="n">AirflowPlugin</span>
+
+<span class="kn">from</span> <span class="nn">flask</span> <span class="k">import</span> <span class="n">Blueprint</span>
+<span class="kn">from</span> <span class="nn">flask_admin</span> <span class="k">import</span> <span class="n">BaseView</span><span class="p">,</span> <span class="n">expose</span>
+<span class="kn">from</span> <span class="nn">flask_admin.base</span> <span class="k">import</span> <span class="n">MenuLink</span>
+
+<span class="c1"># Importing base classes that we need to derive</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="k">import</span> <span class="n">BaseHook</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.executors.base_executor</span> <span class="k">import</span> <span class="n">BaseExecutor</span>
+
+<span class="c1"># Will show up under airflow.hooks.PluginHook</span>
+<span class="k">class</span> <span class="nc">PluginHook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
+ <span class="k">pass</span>
+
+<span class="c1"># Will show up under airflow.operators.PluginOperator</span>
+<span class="k">class</span> <span class="nc">PluginOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="k">pass</span>
+
+<span class="c1"># Will show up under airflow.executors.PluginExecutor</span>
+<span class="k">class</span> <span class="nc">PluginExecutor</span><span class="p">(</span><span class="n">BaseExecutor</span><span class="p">):</span>
+ <span class="k">pass</span>
+
+<span class="c1"># Creating a flask admin BaseView</span>
+<span class="k">class</span> <span class="nc">TestView</span><span class="p">(</span><span class="n">BaseView</span><span class="p">):</span>
+ <span class="nd">@expose</span><span class="p">(</span><span class="s1">'/'</span><span class="p">)</span>
+ <span class="k">def</span> <span class="nf">test</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="c1"># in this example, put your test_plugin/test.html template at airflow/plugins/templates/test_plugin/test.html</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">render</span><span class="p">(</span><span class="s2">"test_plugin/test.html"</span><span class="p">,</span> <span class="n">content</span><span class="o">=</span><span class="s2">"Hello galaxy!"</span><span class="p">)</span>
+<span class="n">v</span> <span class="o">=</span> <span class="n">TestView</span><span class="p">(</span><span class="n">category</span><span class="o">=</span><span class="s2">"Test Plugin"</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s2">"Test View"</span><span class="p">)</span>
+
+<span class="c1"># Creating a flask blueprint to intergrate the templates and static folder</span>
+<span class="n">bp</span> <span class="o">=</span> <span class="n">Blueprint</span><span class="p">(</span>
+ <span class="s2">"test_plugin"</span><span class="p">,</span> <span class="n">__name__</span><span class="p">,</span>
+ <span class="n">template_folder</span><span class="o">=</span><span class="s1">'templates'</span><span class="p">,</span> <span class="c1"># registers airflow/plugins/templates as a Jinja template folder</span>
+ <span class="n">static_folder</span><span class="o">=</span><span class="s1">'static'</span><span class="p">,</span>
+ <span class="n">static_url_path</span><span class="o">=</span><span class="s1">'/static/test_plugin'</span><span class="p">)</span>
+
+<span class="n">ml</span> <span class="o">=</span> <span class="n">MenuLink</span><span class="p">(</span>
+ <span class="n">category</span><span class="o">=</span><span class="s1">'Test Plugin'</span><span class="p">,</span>
+ <span class="n">name</span><span class="o">=</span><span class="s1">'Test Menu Link'</span><span class="p">,</span>
+ <span class="n">url</span><span class="o">=</span><span class="s1">'http://pythonhosted.org/airflow/'</span><span class="p">)</span>
+
+<span class="c1"># Defining the plugin class</span>
+<span class="k">class</span> <span class="nc">AirflowTestPlugin</span><span class="p">(</span><span class="n">AirflowPlugin</span><span class="p">):</span>
+ <span class="n">name</span> <span class="o">=</span> <span class="s2">"test_plugin"</span>
+ <span class="n">operators</span> <span class="o">=</span> <span class="p">[</span><span class="n">PluginOperator</span><span class="p">]</span>
+ <span class="n">flask_blueprints</span> <span class="o">=</span> <span class="p">[</span><span class="n">bp</span><span class="p">]</span>
+ <span class="n">hooks</span> <span class="o">=</span> <span class="p">[</span><span class="n">PluginHook</span><span class="p">]</span>
+ <span class="n">executors</span> <span class="o">=</span> <span class="p">[</span><span class="n">PluginExecutor</span><span class="p">]</span>
+ <span class="n">admin_views</span> <span class="o">=</span> <span class="p">[</span><span class="n">v</span><span class="p">]</span>
+ <span class="n">menu_links</span> <span class="o">=</span> <span class="p">[</span><span class="n">ml</span><span class="p">]</span>
+</pre></div>
+</div>
+</div>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="security.html" class="btn btn-neutral float-right" title="Security" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="scheduler.html" class="btn btn-neutral" title="Scheduling & Triggers" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/profiling.html
----------------------------------------------------------------------
diff --git a/profiling.html b/profiling.html
new file mode 100644
index 0000000..e206ca1
--- /dev/null
+++ b/profiling.html
@@ -0,0 +1,250 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Data Profiling — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="Command Line Interface" href="cli.html"/>
+ <link rel="prev" title="Concepts" href="concepts.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">Data Profiling</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#adhoc-queries">Adhoc Queries</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#charts">Charts</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#chart-screenshot">Chart Screenshot</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#chart-form-screenshot">Chart Form Screenshot</a></li>
+</ul>
+</li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>Data Profiling</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/profiling.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="data-profiling">
+<h1>Data Profiling<a class="headerlink" href="#data-profiling" title="Permalink to this headline">�</a></h1>
+<p>Part of being productive with data is having the right weapons to
+profile the data you are working with. Airflow provides a simple query
+interface to write SQL and get results quickly, and a charting application
+letting you visualize data.</p>
+<div class="section" id="adhoc-queries">
+<h2>Adhoc Queries<a class="headerlink" href="#adhoc-queries" title="Permalink to this headline">�</a></h2>
+<p>The adhoc query UI allows for simple SQL interactions with the database
+connections registered in Airflow.</p>
+<img alt="_images/adhoc.png" src="_images/adhoc.png" />
+</div>
+<div class="section" id="charts">
+<h2>Charts<a class="headerlink" href="#charts" title="Permalink to this headline">�</a></h2>
+<p>A simple UI built on top of flask-admin and highcharts allows building
+data visualizations and charts easily. Fill in a form with a label, SQL,
+chart type, pick a source database from your environment’s connectons,
+select a few other options, and save it for later use.</p>
+<p>You can even use the same templating and macros available when writing
+airflow pipelines, parameterizing your queries and modifying parameters
+directly in the URL.</p>
+<p>These charts are basic, but they’re easy to create, modify and share.</p>
+<div class="section" id="chart-screenshot">
+<h3>Chart Screenshot<a class="headerlink" href="#chart-screenshot" title="Permalink to this headline">�</a></h3>
+<img alt="_images/chart.png" src="_images/chart.png" />
+</div>
+<hr class="docutils" />
+<div class="section" id="chart-form-screenshot">
+<h3>Chart Form Screenshot<a class="headerlink" href="#chart-form-screenshot" title="Permalink to this headline">�</a></h3>
+<img alt="_images/chart_form.png" src="_images/chart_form.png" />
+</div>
+</div>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="cli.html" class="btn btn-neutral float-right" title="Command Line Interface" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="concepts.html" class="btn btn-neutral" title="Concepts" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/project.html
----------------------------------------------------------------------
diff --git a/project.html b/project.html
new file mode 100644
index 0000000..4c8b915
--- /dev/null
+++ b/project.html
@@ -0,0 +1,268 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Project — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="License" href="license.html"/>
+ <link rel="prev" title="Apache Airflow (incubating) Documentation" href="index.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1 current"><a class="current reference internal" href="#">Project</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#history">History</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#committers">Committers</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#resources-links">Resources & links</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#roadmap">Roadmap</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>Project</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/project.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="project">
+<h1>Project<a class="headerlink" href="#project" title="Permalink to this headline">�</a></h1>
+<div class="section" id="history">
+<h2>History<a class="headerlink" href="#history" title="Permalink to this headline">�</a></h2>
+<p>Airflow was started in the fall of 2014 by Maxime Beauchemin at Airbnb.
+It was open source from the very first commit and officially brought under
+the Airbnb Github and announced in the spring of 2015.</p>
+<p>The project joined the Apache Software Foundation’s incubation program in the
+winter of 2016.</p>
+</div>
+<div class="section" id="committers">
+<h2>Committers<a class="headerlink" href="#committers" title="Permalink to this headline">�</a></h2>
+<ul class="simple">
+<li>@mistercrunch (Maxime “Max” Beauchemin)</li>
+<li>@r39132 (Siddharth “Sid” Anand)</li>
+<li>@criccomini (Chris Riccomini)</li>
+<li>@bolkedebruin (Bolke de Bruin)</li>
+<li>@artwr (Arthur Wiedmer)</li>
+<li>@jlowin (Jeremiah Lowin)</li>
+<li>@patrickleotardif (Patrick Leo Tardif)</li>
+<li>@aoen (Dan Davydov)</li>
+</ul>
+<p>For the full list of contributors, take a look at <a class="reference external" href="https://github.com/apache/incubator-airflow/graphs/contributors">Airflow’s Github
+Contributor page:</a></p>
+</div>
+<div class="section" id="resources-links">
+<h2>Resources & links<a class="headerlink" href="#resources-links" title="Permalink to this headline">�</a></h2>
+<ul class="simple">
+<li>Mailing list (send emails to
+<code class="docutils literal"><span class="pre">dev-subscribe@airflow.incubator.apache.org</span></code> and
+<code class="docutils literal"><span class="pre">commits-subscribe@airflow.incubator.apache.org</span></code>
+to subscribe to each)</li>
+<li><a class="reference external" href="https://issues.apache.org/jira/browse/AIRFLOW">Issues</a></li>
+<li><a class="reference external" href="http://nerds.airbnb.com/airflow/">Airbnb Blog Post about Airflow</a></li>
+<li><a class="reference external" href="https://cwiki.apache.org/confluence/display/AIRFLOW/Common+Pitfalls">Airflow Common Pitfalls</a></li>
+<li><a class="reference external" href="https://www.youtube.com/watch?v=oYp49mBwH60">Hadoop Summit Airflow Video</a></li>
+<li><a class="reference external" href="http://agari.com/blog/airflow-agari">Airflow at Agari Blog Post</a></li>
+<li><a class="reference external" href="https://youtu.be/dgaoqOZlvEA">Talk: Best practices with Airflow (nov 2015)</a></li>
+<li><a class="reference external" href="https://www.linkedin.com/pulse/airflow-lesson-1-triggerdagrunoperator-siddharth-anand?published=t">Airflow Lesson 1: TriggerDagRunOperator</a></li>
+<li><a class="reference external" href="https://github.com/puckel/docker-airflow">Docker Airflow (externally maintained)</a></li>
+<li><a class="reference external" href="https://medium.com/handy-tech/airflow-tips-tricks-and-pitfalls-9ba53fba14eb#.o2snqeoz7">Airflow: Tips, Tricks, and Pitfalls @ Handy</a></li>
+<li><a class="reference external" href="https://github.com/bahchis/airflow-cookbook">Airflow Chef recipe (community contributed)</a> ,
+<a class="reference external" href="https://supermarket.chef.io/cookbooks/airflow">another here</a></li>
+<li>Airflow Puppet Module (community contributed) <a class="reference external" href="https://github.com/similarweb/puppet-airflow">puppet-airflow</a>,
+<a class="reference external" href="https://forge.puppetlabs.com/similarweb/airflow">airflow</a></li>
+<li><a class="reference external" href="https://gitter.im/airbnb/airflow">Gitter (chat) Channel</a></li>
+</ul>
+</div>
+<div class="section" id="roadmap">
+<h2>Roadmap<a class="headerlink" href="#roadmap" title="Permalink to this headline">�</a></h2>
+<p>Please refer to the Roadmap on <a class="reference external" href="https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Home">the wiki</a></p>
+</div>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="license.html" class="btn btn-neutral float-right" title="License" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="index.html" class="btn btn-neutral" title="Apache Airflow (incubating) Documentation" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/py-modindex.html
----------------------------------------------------------------------
diff --git a/py-modindex.html b/py-modindex.html
new file mode 100644
index 0000000..94ed176
--- /dev/null
+++ b/py-modindex.html
@@ -0,0 +1,262 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Python Module Index — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+
+
+
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li></li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+
+ <h1>Python Module Index</h1>
+
+ <div class="modindex-jumpbox">
+ <a href="#cap-a"><strong>a</strong></a>
+ </div>
+
+ <table class="indextable modindextable" cellspacing="0" cellpadding="2">
+ <tr class="pcap"><td></td><td> </td><td></td></tr>
+ <tr class="cap" id="cap-a"><td></td><td>
+ <strong>a</strong></td><td></td></tr>
+ <tr>
+ <td><img src="_static/minus.png" class="toggler"
+ id="toggle-1" style="display: none" alt="-" /></td>
+ <td>
+ <code class="xref">airflow</code></td><td>
+ <em></em></td></tr>
+ <tr class="cg-1">
+ <td></td>
+ <td>
+ <a href="code.html#module-airflow.contrib.executors"><code class="xref">airflow.contrib.executors</code></a></td><td>
+ <em></em></td></tr>
+ <tr class="cg-1">
+ <td></td>
+ <td>
+ <a href="code.html#module-airflow.contrib.hooks"><code class="xref">airflow.contrib.hooks</code></a></td><td>
+ <em></em></td></tr>
+ <tr class="cg-1">
+ <td></td>
+ <td>
+ <a href="code.html#module-airflow.contrib.operators"><code class="xref">airflow.contrib.operators</code></a></td><td>
+ <em></em></td></tr>
+ <tr class="cg-1">
+ <td></td>
+ <td>
+ <a href="code.html#module-airflow.executors"><code class="xref">airflow.executors</code></a></td><td>
+ <em></em></td></tr>
+ <tr class="cg-1">
+ <td></td>
+ <td>
+ <a href="code.html#module-airflow.hooks"><code class="xref">airflow.hooks</code></a></td><td>
+ <em></em></td></tr>
+ <tr class="cg-1">
+ <td></td>
+ <td>
+ <a href="code.html#module-airflow.macros"><code class="xref">airflow.macros</code></a></td><td>
+ <em></em></td></tr>
+ <tr class="cg-1">
+ <td></td>
+ <td>
+ <a href="code.html#module-airflow.macros.hive"><code class="xref">airflow.macros.hive</code></a></td><td>
+ <em></em></td></tr>
+ <tr class="cg-1">
+ <td></td>
+ <td>
+ <a href="code.html#module-airflow.models"><code class="xref">airflow.models</code></a></td><td>
+ <em></em></td></tr>
+ <tr class="cg-1">
+ <td></td>
+ <td>
+ <a href="code.html#module-airflow.operators"><code class="xref">airflow.operators</code></a></td><td>
+ <em></em></td></tr>
+ </table>
+
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[16/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/basic.css
----------------------------------------------------------------------
diff --git a/_static/basic.css b/_static/basic.css
new file mode 100644
index 0000000..65dfd7d
--- /dev/null
+++ b/_static/basic.css
@@ -0,0 +1,608 @@
+/*
+ * basic.css
+ * ~~~~~~~~~
+ *
+ * Sphinx stylesheet -- basic theme.
+ *
+ * :copyright: Copyright 2007-2016 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+/* -- main layout ----------------------------------------------------------- */
+
+div.clearer {
+ clear: both;
+}
+
+/* -- relbar ---------------------------------------------------------------- */
+
+div.related {
+ width: 100%;
+ font-size: 90%;
+}
+
+div.related h3 {
+ display: none;
+}
+
+div.related ul {
+ margin: 0;
+ padding: 0 0 0 10px;
+ list-style: none;
+}
+
+div.related li {
+ display: inline;
+}
+
+div.related li.right {
+ float: right;
+ margin-right: 5px;
+}
+
+/* -- sidebar --------------------------------------------------------------- */
+
+div.sphinxsidebarwrapper {
+ padding: 10px 5px 0 10px;
+}
+
+div.sphinxsidebar {
+ float: left;
+ width: 230px;
+ margin-left: -100%;
+ font-size: 90%;
+ word-wrap: break-word;
+ overflow-wrap : break-word;
+}
+
+div.sphinxsidebar ul {
+ list-style: none;
+}
+
+div.sphinxsidebar ul ul,
+div.sphinxsidebar ul.want-points {
+ margin-left: 20px;
+ list-style: square;
+}
+
+div.sphinxsidebar ul ul {
+ margin-top: 0;
+ margin-bottom: 0;
+}
+
+div.sphinxsidebar form {
+ margin-top: 10px;
+}
+
+div.sphinxsidebar input {
+ border: 1px solid #98dbcc;
+ font-family: sans-serif;
+ font-size: 1em;
+}
+
+div.sphinxsidebar #searchbox input[type="text"] {
+ width: 170px;
+}
+
+div.sphinxsidebar #searchbox input[type="submit"] {
+ width: 30px;
+}
+
+img {
+ border: 0;
+ max-width: 100%;
+}
+
+/* -- search page ----------------------------------------------------------- */
+
+ul.search {
+ margin: 10px 0 0 20px;
+ padding: 0;
+}
+
+ul.search li {
+ padding: 5px 0 5px 20px;
+ background-image: url(file.png);
+ background-repeat: no-repeat;
+ background-position: 0 7px;
+}
+
+ul.search li a {
+ font-weight: bold;
+}
+
+ul.search li div.context {
+ color: #888;
+ margin: 2px 0 0 30px;
+ text-align: left;
+}
+
+ul.keywordmatches li.goodmatch a {
+ font-weight: bold;
+}
+
+/* -- index page ------------------------------------------------------------ */
+
+table.contentstable {
+ width: 90%;
+}
+
+table.contentstable p.biglink {
+ line-height: 150%;
+}
+
+a.biglink {
+ font-size: 1.3em;
+}
+
+span.linkdescr {
+ font-style: italic;
+ padding-top: 5px;
+ font-size: 90%;
+}
+
+/* -- general index --------------------------------------------------------- */
+
+table.indextable {
+ width: 100%;
+}
+
+table.indextable td {
+ text-align: left;
+ vertical-align: top;
+}
+
+table.indextable dl, table.indextable dd {
+ margin-top: 0;
+ margin-bottom: 0;
+}
+
+table.indextable tr.pcap {
+ height: 10px;
+}
+
+table.indextable tr.cap {
+ margin-top: 10px;
+ background-color: #f2f2f2;
+}
+
+img.toggler {
+ margin-right: 3px;
+ margin-top: 3px;
+ cursor: pointer;
+}
+
+div.modindex-jumpbox {
+ border-top: 1px solid #ddd;
+ border-bottom: 1px solid #ddd;
+ margin: 1em 0 1em 0;
+ padding: 0.4em;
+}
+
+div.genindex-jumpbox {
+ border-top: 1px solid #ddd;
+ border-bottom: 1px solid #ddd;
+ margin: 1em 0 1em 0;
+ padding: 0.4em;
+}
+
+/* -- general body styles --------------------------------------------------- */
+
+div.body p, div.body dd, div.body li, div.body blockquote {
+ -moz-hyphens: auto;
+ -ms-hyphens: auto;
+ -webkit-hyphens: auto;
+ hyphens: auto;
+}
+
+a.headerlink {
+ visibility: hidden;
+}
+
+h1:hover > a.headerlink,
+h2:hover > a.headerlink,
+h3:hover > a.headerlink,
+h4:hover > a.headerlink,
+h5:hover > a.headerlink,
+h6:hover > a.headerlink,
+dt:hover > a.headerlink,
+caption:hover > a.headerlink,
+p.caption:hover > a.headerlink,
+div.code-block-caption:hover > a.headerlink {
+ visibility: visible;
+}
+
+div.body p.caption {
+ text-align: inherit;
+}
+
+div.body td {
+ text-align: left;
+}
+
+.field-list ul {
+ padding-left: 1em;
+}
+
+.first {
+ margin-top: 0 !important;
+}
+
+p.rubric {
+ margin-top: 30px;
+ font-weight: bold;
+}
+
+img.align-left, .figure.align-left, object.align-left {
+ clear: left;
+ float: left;
+ margin-right: 1em;
+}
+
+img.align-right, .figure.align-right, object.align-right {
+ clear: right;
+ float: right;
+ margin-left: 1em;
+}
+
+img.align-center, .figure.align-center, object.align-center {
+ display: block;
+ margin-left: auto;
+ margin-right: auto;
+}
+
+.align-left {
+ text-align: left;
+}
+
+.align-center {
+ text-align: center;
+}
+
+.align-right {
+ text-align: right;
+}
+
+/* -- sidebars -------------------------------------------------------------- */
+
+div.sidebar {
+ margin: 0 0 0.5em 1em;
+ border: 1px solid #ddb;
+ padding: 7px 7px 0 7px;
+ background-color: #ffe;
+ width: 40%;
+ float: right;
+}
+
+p.sidebar-title {
+ font-weight: bold;
+}
+
+/* -- topics ---------------------------------------------------------------- */
+
+div.topic {
+ border: 1px solid #ccc;
+ padding: 7px 7px 0 7px;
+ margin: 10px 0 10px 0;
+}
+
+p.topic-title {
+ font-size: 1.1em;
+ font-weight: bold;
+ margin-top: 10px;
+}
+
+/* -- admonitions ----------------------------------------------------------- */
+
+div.admonition {
+ margin-top: 10px;
+ margin-bottom: 10px;
+ padding: 7px;
+}
+
+div.admonition dt {
+ font-weight: bold;
+}
+
+div.admonition dl {
+ margin-bottom: 0;
+}
+
+p.admonition-title {
+ margin: 0px 10px 5px 0px;
+ font-weight: bold;
+}
+
+div.body p.centered {
+ text-align: center;
+ margin-top: 25px;
+}
+
+/* -- tables ---------------------------------------------------------------- */
+
+table.docutils {
+ border: 0;
+ border-collapse: collapse;
+}
+
+table caption span.caption-number {
+ font-style: italic;
+}
+
+table caption span.caption-text {
+}
+
+table.docutils td, table.docutils th {
+ padding: 1px 8px 1px 5px;
+ border-top: 0;
+ border-left: 0;
+ border-right: 0;
+ border-bottom: 1px solid #aaa;
+}
+
+table.field-list td, table.field-list th {
+ border: 0 !important;
+}
+
+table.footnote td, table.footnote th {
+ border: 0 !important;
+}
+
+th {
+ text-align: left;
+ padding-right: 5px;
+}
+
+table.citation {
+ border-left: solid 1px gray;
+ margin-left: 1px;
+}
+
+table.citation td {
+ border-bottom: none;
+}
+
+/* -- figures --------------------------------------------------------------- */
+
+div.figure {
+ margin: 0.5em;
+ padding: 0.5em;
+}
+
+div.figure p.caption {
+ padding: 0.3em;
+}
+
+div.figure p.caption span.caption-number {
+ font-style: italic;
+}
+
+div.figure p.caption span.caption-text {
+}
+
+
+/* -- other body styles ----------------------------------------------------- */
+
+ol.arabic {
+ list-style: decimal;
+}
+
+ol.loweralpha {
+ list-style: lower-alpha;
+}
+
+ol.upperalpha {
+ list-style: upper-alpha;
+}
+
+ol.lowerroman {
+ list-style: lower-roman;
+}
+
+ol.upperroman {
+ list-style: upper-roman;
+}
+
+dl {
+ margin-bottom: 15px;
+}
+
+dd p {
+ margin-top: 0px;
+}
+
+dd ul, dd table {
+ margin-bottom: 10px;
+}
+
+dd {
+ margin-top: 3px;
+ margin-bottom: 10px;
+ margin-left: 30px;
+}
+
+dt:target, .highlighted {
+ background-color: #fbe54e;
+}
+
+dl.glossary dt {
+ font-weight: bold;
+ font-size: 1.1em;
+}
+
+.field-list ul {
+ margin: 0;
+ padding-left: 1em;
+}
+
+.field-list p {
+ margin: 0;
+}
+
+.optional {
+ font-size: 1.3em;
+}
+
+.sig-paren {
+ font-size: larger;
+}
+
+.versionmodified {
+ font-style: italic;
+}
+
+.system-message {
+ background-color: #fda;
+ padding: 5px;
+ border: 3px solid red;
+}
+
+.footnote:target {
+ background-color: #ffa;
+}
+
+.line-block {
+ display: block;
+ margin-top: 1em;
+ margin-bottom: 1em;
+}
+
+.line-block .line-block {
+ margin-top: 0;
+ margin-bottom: 0;
+ margin-left: 1.5em;
+}
+
+.guilabel, .menuselection {
+ font-family: sans-serif;
+}
+
+.accelerator {
+ text-decoration: underline;
+}
+
+.classifier {
+ font-style: oblique;
+}
+
+abbr, acronym {
+ border-bottom: dotted 1px;
+ cursor: help;
+}
+
+/* -- code displays --------------------------------------------------------- */
+
+pre {
+ overflow: auto;
+ overflow-y: hidden; /* fixes display issues on Chrome browsers */
+}
+
+td.linenos pre {
+ padding: 5px 0px;
+ border: 0;
+ background-color: transparent;
+ color: #aaa;
+}
+
+table.highlighttable {
+ margin-left: 0.5em;
+}
+
+table.highlighttable td {
+ padding: 0 0.5em 0 0.5em;
+}
+
+div.code-block-caption {
+ padding: 2px 5px;
+ font-size: small;
+}
+
+div.code-block-caption code {
+ background-color: transparent;
+}
+
+div.code-block-caption + div > div.highlight > pre {
+ margin-top: 0;
+}
+
+div.code-block-caption span.caption-number {
+ padding: 0.1em 0.3em;
+ font-style: italic;
+}
+
+div.code-block-caption span.caption-text {
+}
+
+div.literal-block-wrapper {
+ padding: 1em 1em 0;
+}
+
+div.literal-block-wrapper div.highlight {
+ margin: 0;
+}
+
+code.descname {
+ background-color: transparent;
+ font-weight: bold;
+ font-size: 1.2em;
+}
+
+code.descclassname {
+ background-color: transparent;
+}
+
+code.xref, a code {
+ background-color: transparent;
+ font-weight: bold;
+}
+
+h1 code, h2 code, h3 code, h4 code, h5 code, h6 code {
+ background-color: transparent;
+}
+
+.viewcode-link {
+ float: right;
+}
+
+.viewcode-back {
+ float: right;
+ font-family: sans-serif;
+}
+
+div.viewcode-block:target {
+ margin: -1px -10px;
+ padding: 0 10px;
+}
+
+/* -- math display ---------------------------------------------------------- */
+
+img.math {
+ vertical-align: middle;
+}
+
+div.body div.math p {
+ text-align: center;
+}
+
+span.eqno {
+ float: right;
+}
+
+/* -- printout stylesheet --------------------------------------------------- */
+
+@media print {
+ div.document,
+ div.documentwrapper,
+ div.bodywrapper {
+ margin: 0 !important;
+ width: 100%;
+ }
+
+ div.sphinxsidebar,
+ div.related,
+ div.footer,
+ #top-link {
+ display: none;
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/comment-bright.png
----------------------------------------------------------------------
diff --git a/_static/comment-bright.png b/_static/comment-bright.png
new file mode 100644
index 0000000..551517b
Binary files /dev/null and b/_static/comment-bright.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/comment-close.png
----------------------------------------------------------------------
diff --git a/_static/comment-close.png b/_static/comment-close.png
new file mode 100644
index 0000000..09b54be
Binary files /dev/null and b/_static/comment-close.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/comment.png
----------------------------------------------------------------------
diff --git a/_static/comment.png b/_static/comment.png
new file mode 100644
index 0000000..92feb52
Binary files /dev/null and b/_static/comment.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/css/badge_only.css
----------------------------------------------------------------------
diff --git a/_static/css/badge_only.css b/_static/css/badge_only.css
new file mode 100644
index 0000000..7e17fb1
--- /dev/null
+++ b/_static/css/badge_only.css
@@ -0,0 +1,2 @@
+\ufeff.fa:before{-webkit-font-smoothing:antialiased}.clearfix{*zoom:1}.clearfix:before,.clearfix:after{display:table;content:""}.clearfix:after{clear:both}@font-face{font-family:FontAwesome;font-weight:normal;font-style:normal;src:url("../font/fontawesome_webfont.eot");src:url("../font/fontawesome_webfont.eot?#iefix") format("embedded-opentype"),url("../font/fontawesome_webfont.woff") format("woff"),url("../font/fontawesome_webfont.ttf") format("truetype"),url("../font/fontawesome_webfont.svg#FontAwesome") format("svg")}.fa:before{display:inline-block;font-family:FontAwesome;font-style:normal;font-weight:normal;line-height:1;text-decoration:inherit}a .fa{display:inline-block;text-decoration:inherit}li .fa{display:inline-block}li .fa-large:before,li .fa-large:before{width:1.875em}ul.fas{list-style-type:none;margin-left:2em;text-indent:-0.8em}ul.fas li .fa{width:0.8em}ul.fas li .fa-large:before,ul.fas li .fa-large:before{vertical-align:baseline}.fa-book:before{content:"\uf02d"}.icon-book:be
fore{content:"\uf02d"}.fa-caret-down:before{content:"\uf0d7"}.icon-caret-down:before{content:"\uf0d7"}.fa-caret-up:before{content:"\uf0d8"}.icon-caret-up:before{content:"\uf0d8"}.fa-caret-left:before{content:"\uf0d9"}.icon-caret-left:before{content:"\uf0d9"}.fa-caret-right:before{content:"\uf0da"}.icon-caret-right:before{content:"\uf0da"}.rst-versions{position:fixed;bottom:0;left:0;width:300px;color:#fcfcfc;background:#1f1d1d;border-top:solid 10px #343131;font-family:"Lato","proxima-nova","Helvetica Neue",Arial,sans-serif;z-index:400}.rst-versions a{color:#2980B9;text-decoration:none}.rst-versions .rst-badge-small{display:none}.rst-versions .rst-current-version{padding:12px;background-color:#272525;display:block;text-align:right;font-size:90%;cursor:pointer;color:#27AE60;*zoom:1}.rst-versions .rst-current-version:before,.rst-versions .rst-current-version:after{display:table;content:""}.rst-versions .rst-current-version:after{clear:both}.rst-versions .rst-current-version .fa{color:#fcfcfc}.rst-versions .rst-cu
rrent-version .fa-book{float:left}.rst-versions .rst-current-version .icon-book{float:left}.rst-versions .rst-current-version.rst-out-of-date{background-color:#E74C3C;color:#fff}.rst-versions .rst-current-version.rst-active-old-version{background-color:#F1C40F;color:#000}.rst-versions.shift-up .rst-other-versions{display:block}.rst-versions .rst-other-versions{font-size:90%;padding:12px;color:gray;display:none}.rst-versions .rst-other-versions hr{display:block;height:1px;border:0;margin:20px 0;padding:0;border-top:solid 1px #413d3d}.rst-versions .rst-other-versions dd{display:inline-block;margin:0}.rst-versions .rst-other-versions dd a{display:inline-block;padding:6px;color:#fcfcfc}.rst-versions.rst-badge{width:auto;bottom:20px;right:20px;left:auto;border:none;max-width:300px}.rst-versions.rst-badge .icon-book{float:none}.rst-versions.rst-badge .fa-book{float:none}.rst-versions.rst-badge.shift-up .rst-current-version{text-align:right}.rst-versions.rst-badge.shift-up .rst-current-ver
sion .fa-book{float:left}.rst-versions.rst-badge.shift-up .rst-current-version .icon-book{float:left}.rst-versions.rst-badge .rst-current-version{width:auto;height:30px;line-height:30px;padding:0 6px;display:block;text-align:center}@media screen and (max-width: 768px){.rst-versions{width:85%;display:none}.rst-versions.shift{display:block}img{width:100%;height:auto}}
+/*# sourceMappingURL=badge_only.css.map */
[13/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/fonts/fontawesome-webfont.svg
----------------------------------------------------------------------
diff --git a/_static/fonts/fontawesome-webfont.svg b/_static/fonts/fontawesome-webfont.svg
new file mode 100644
index 0000000..d907b25
--- /dev/null
+++ b/_static/fonts/fontawesome-webfont.svg
@@ -0,0 +1,520 @@
+<?xml version="1.0" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd" >
+<svg xmlns="http://www.w3.org/2000/svg">
+<metadata></metadata>
+<defs>
+<font id="fontawesomeregular" horiz-adv-x="1536" >
+<font-face units-per-em="1792" ascent="1536" descent="-256" />
+<missing-glyph horiz-adv-x="448" />
+<glyph unicode=" " horiz-adv-x="448" />
+<glyph unicode="	" horiz-adv-x="448" />
+<glyph unicode=" " horiz-adv-x="448" />
+<glyph unicode="¨" horiz-adv-x="1792" />
+<glyph unicode="©" horiz-adv-x="1792" />
+<glyph unicode="®" horiz-adv-x="1792" />
+<glyph unicode="´" horiz-adv-x="1792" />
+<glyph unicode="Æ" horiz-adv-x="1792" />
+<glyph unicode="Ø" horiz-adv-x="1792" />
+<glyph unicode=" " horiz-adv-x="768" />
+<glyph unicode=" " horiz-adv-x="1537" />
+<glyph unicode=" " horiz-adv-x="768" />
+<glyph unicode=" " horiz-adv-x="1537" />
+<glyph unicode=" " horiz-adv-x="512" />
+<glyph unicode=" " horiz-adv-x="384" />
+<glyph unicode=" " horiz-adv-x="256" />
+<glyph unicode=" " horiz-adv-x="256" />
+<glyph unicode=" " horiz-adv-x="192" />
+<glyph unicode=" " horiz-adv-x="307" />
+<glyph unicode=" " horiz-adv-x="85" />
+<glyph unicode=" " horiz-adv-x="307" />
+<glyph unicode=" " horiz-adv-x="384" />
+<glyph unicode="™" horiz-adv-x="1792" />
+<glyph unicode="∞" horiz-adv-x="1792" />
+<glyph unicode="≠" horiz-adv-x="1792" />
+<glyph unicode="◼" horiz-adv-x="500" d="M0 0z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1699 1350q0 -35 -43 -78l-632 -632v-768h320q26 0 45 -19t19 -45t-19 -45t-45 -19h-896q-26 0 -45 19t-19 45t19 45t45 19h320v768l-632 632q-43 43 -43 78q0 23 18 36.5t38 17.5t43 4h1408q23 0 43 -4t38 -17.5t18 -36.5z" />
+<glyph unicode="" d="M1536 1312v-1120q0 -50 -34 -89t-86 -60.5t-103.5 -32t-96.5 -10.5t-96.5 10.5t-103.5 32t-86 60.5t-34 89t34 89t86 60.5t103.5 32t96.5 10.5q105 0 192 -39v537l-768 -237v-709q0 -50 -34 -89t-86 -60.5t-103.5 -32t-96.5 -10.5t-96.5 10.5t-103.5 32t-86 60.5t-34 89 t34 89t86 60.5t103.5 32t96.5 10.5q105 0 192 -39v967q0 31 19 56.5t49 35.5l832 256q12 4 28 4q40 0 68 -28t28 -68z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1152 704q0 185 -131.5 316.5t-316.5 131.5t-316.5 -131.5t-131.5 -316.5t131.5 -316.5t316.5 -131.5t316.5 131.5t131.5 316.5zM1664 -128q0 -52 -38 -90t-90 -38q-54 0 -90 38l-343 342q-179 -124 -399 -124q-143 0 -273.5 55.5t-225 150t-150 225t-55.5 273.5 t55.5 273.5t150 225t225 150t273.5 55.5t273.5 -55.5t225 -150t150 -225t55.5 -273.5q0 -220 -124 -399l343 -343q37 -37 37 -90z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1664 32v768q-32 -36 -69 -66q-268 -206 -426 -338q-51 -43 -83 -67t-86.5 -48.5t-102.5 -24.5h-1h-1q-48 0 -102.5 24.5t-86.5 48.5t-83 67q-158 132 -426 338q-37 30 -69 66v-768q0 -13 9.5 -22.5t22.5 -9.5h1472q13 0 22.5 9.5t9.5 22.5zM1664 1083v11v13.5t-0.5 13 t-3 12.5t-5.5 9t-9 7.5t-14 2.5h-1472q-13 0 -22.5 -9.5t-9.5 -22.5q0 -168 147 -284q193 -152 401 -317q6 -5 35 -29.5t46 -37.5t44.5 -31.5t50.5 -27.5t43 -9h1h1q20 0 43 9t50.5 27.5t44.5 31.5t46 37.5t35 29.5q208 165 401 317q54 43 100.5 115.5t46.5 131.5z M1792 1120v-1088q0 -66 -47 -113t-113 -47h-1472q-66 0 -113 47t-47 113v1088q0 66 47 113t113 47h1472q66 0 113 -47t47 -113z" />
+<glyph unicode="" horiz-adv-x="1792" d="M896 -128q-26 0 -44 18l-624 602q-10 8 -27.5 26t-55.5 65.5t-68 97.5t-53.5 121t-23.5 138q0 220 127 344t351 124q62 0 126.5 -21.5t120 -58t95.5 -68.5t76 -68q36 36 76 68t95.5 68.5t120 58t126.5 21.5q224 0 351 -124t127 -344q0 -221 -229 -450l-623 -600 q-18 -18 -44 -18z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1664 889q0 -22 -26 -48l-363 -354l86 -500q1 -7 1 -20q0 -21 -10.5 -35.5t-30.5 -14.5q-19 0 -40 12l-449 236l-449 -236q-22 -12 -40 -12q-21 0 -31.5 14.5t-10.5 35.5q0 6 2 20l86 500l-364 354q-25 27 -25 48q0 37 56 46l502 73l225 455q19 41 49 41t49 -41l225 -455 l502 -73q56 -9 56 -46z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1137 532l306 297l-422 62l-189 382l-189 -382l-422 -62l306 -297l-73 -421l378 199l377 -199zM1664 889q0 -22 -26 -48l-363 -354l86 -500q1 -7 1 -20q0 -50 -41 -50q-19 0 -40 12l-449 236l-449 -236q-22 -12 -40 -12q-21 0 -31.5 14.5t-10.5 35.5q0 6 2 20l86 500 l-364 354q-25 27 -25 48q0 37 56 46l502 73l225 455q19 41 49 41t49 -41l225 -455l502 -73q56 -9 56 -46z" />
+<glyph unicode="" horiz-adv-x="1408" d="M1408 131q0 -120 -73 -189.5t-194 -69.5h-874q-121 0 -194 69.5t-73 189.5q0 53 3.5 103.5t14 109t26.5 108.5t43 97.5t62 81t85.5 53.5t111.5 20q9 0 42 -21.5t74.5 -48t108 -48t133.5 -21.5t133.5 21.5t108 48t74.5 48t42 21.5q61 0 111.5 -20t85.5 -53.5t62 -81 t43 -97.5t26.5 -108.5t14 -109t3.5 -103.5zM1088 1024q0 -159 -112.5 -271.5t-271.5 -112.5t-271.5 112.5t-112.5 271.5t112.5 271.5t271.5 112.5t271.5 -112.5t112.5 -271.5z" />
+<glyph unicode="" horiz-adv-x="1920" d="M384 -64v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM384 320v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM384 704v128q0 26 -19 45t-45 19h-128 q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1408 -64v512q0 26 -19 45t-45 19h-768q-26 0 -45 -19t-19 -45v-512q0 -26 19 -45t45 -19h768q26 0 45 19t19 45zM384 1088v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45 t45 -19h128q26 0 45 19t19 45zM1792 -64v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1408 704v512q0 26 -19 45t-45 19h-768q-26 0 -45 -19t-19 -45v-512q0 -26 19 -45t45 -19h768q26 0 45 19t19 45zM1792 320v128 q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1792 704v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t1
9 45zM1792 1088v128q0 26 -19 45t-45 19h-128q-26 0 -45 -19 t-19 -45v-128q0 -26 19 -45t45 -19h128q26 0 45 19t19 45zM1920 1248v-1344q0 -66 -47 -113t-113 -47h-1600q-66 0 -113 47t-47 113v1344q0 66 47 113t113 47h1600q66 0 113 -47t47 -113z" />
+<glyph unicode="" horiz-adv-x="1664" d="M768 512v-384q0 -52 -38 -90t-90 -38h-512q-52 0 -90 38t-38 90v384q0 52 38 90t90 38h512q52 0 90 -38t38 -90zM768 1280v-384q0 -52 -38 -90t-90 -38h-512q-52 0 -90 38t-38 90v384q0 52 38 90t90 38h512q52 0 90 -38t38 -90zM1664 512v-384q0 -52 -38 -90t-90 -38 h-512q-52 0 -90 38t-38 90v384q0 52 38 90t90 38h512q52 0 90 -38t38 -90zM1664 1280v-384q0 -52 -38 -90t-90 -38h-512q-52 0 -90 38t-38 90v384q0 52 38 90t90 38h512q52 0 90 -38t38 -90z" />
+<glyph unicode="" horiz-adv-x="1792" d="M512 288v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM512 800v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1152 288v-192q0 -40 -28 -68t-68 -28h-320 q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM512 1312v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1152 800v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28 h320q40 0 68 -28t28 -68zM1792 288v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1152 1312v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1792 800v-192 q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1792 1312v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28
t28 -68z" />
+<glyph unicode="" horiz-adv-x="1792" d="M512 288v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM512 800v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1792 288v-192q0 -40 -28 -68t-68 -28h-960 q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h960q40 0 68 -28t28 -68zM512 1312v-192q0 -40 -28 -68t-68 -28h-320q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h320q40 0 68 -28t28 -68zM1792 800v-192q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v192q0 40 28 68t68 28 h960q40 0 68 -28t28 -68zM1792 1312v-192q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h960q40 0 68 -28t28 -68z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1671 970q0 -40 -28 -68l-724 -724l-136 -136q-28 -28 -68 -28t-68 28l-136 136l-362 362q-28 28 -28 68t28 68l136 136q28 28 68 28t68 -28l294 -295l656 657q28 28 68 28t68 -28l136 -136q28 -28 28 -68z" />
+<glyph unicode="" horiz-adv-x="1408" d="M1298 214q0 -40 -28 -68l-136 -136q-28 -28 -68 -28t-68 28l-294 294l-294 -294q-28 -28 -68 -28t-68 28l-136 136q-28 28 -28 68t28 68l294 294l-294 294q-28 28 -28 68t28 68l136 136q28 28 68 28t68 -28l294 -294l294 294q28 28 68 28t68 -28l136 -136q28 -28 28 -68 t-28 -68l-294 -294l294 -294q28 -28 28 -68z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1024 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-224v-224q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v224h-224q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h224v224q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5v-224h224 q13 0 22.5 -9.5t9.5 -22.5zM1152 704q0 185 -131.5 316.5t-316.5 131.5t-316.5 -131.5t-131.5 -316.5t131.5 -316.5t316.5 -131.5t316.5 131.5t131.5 316.5zM1664 -128q0 -53 -37.5 -90.5t-90.5 -37.5q-54 0 -90 38l-343 342q-179 -124 -399 -124q-143 0 -273.5 55.5 t-225 150t-150 225t-55.5 273.5t55.5 273.5t150 225t225 150t273.5 55.5t273.5 -55.5t225 -150t150 -225t55.5 -273.5q0 -220 -124 -399l343 -343q37 -37 37 -90z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1024 736v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-576q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h576q13 0 22.5 -9.5t9.5 -22.5zM1152 704q0 185 -131.5 316.5t-316.5 131.5t-316.5 -131.5t-131.5 -316.5t131.5 -316.5t316.5 -131.5t316.5 131.5t131.5 316.5z M1664 -128q0 -53 -37.5 -90.5t-90.5 -37.5q-54 0 -90 38l-343 342q-179 -124 -399 -124q-143 0 -273.5 55.5t-225 150t-150 225t-55.5 273.5t55.5 273.5t150 225t225 150t273.5 55.5t273.5 -55.5t225 -150t150 -225t55.5 -273.5q0 -220 -124 -399l343 -343q37 -37 37 -90z " />
+<glyph unicode="" d="M1536 640q0 -156 -61 -298t-164 -245t-245 -164t-298 -61t-298 61t-245 164t-164 245t-61 298q0 182 80.5 343t226.5 270q43 32 95.5 25t83.5 -50q32 -42 24.5 -94.5t-49.5 -84.5q-98 -74 -151.5 -181t-53.5 -228q0 -104 40.5 -198.5t109.5 -163.5t163.5 -109.5 t198.5 -40.5t198.5 40.5t163.5 109.5t109.5 163.5t40.5 198.5q0 121 -53.5 228t-151.5 181q-42 32 -49.5 84.5t24.5 94.5q31 43 84 50t95 -25q146 -109 226.5 -270t80.5 -343zM896 1408v-640q0 -52 -38 -90t-90 -38t-90 38t-38 90v640q0 52 38 90t90 38t90 -38t38 -90z" />
+<glyph unicode="" horiz-adv-x="1792" d="M256 96v-192q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h192q14 0 23 -9t9 -23zM640 224v-320q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v320q0 14 9 23t23 9h192q14 0 23 -9t9 -23zM1024 480v-576q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23 v576q0 14 9 23t23 9h192q14 0 23 -9t9 -23zM1408 864v-960q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v960q0 14 9 23t23 9h192q14 0 23 -9t9 -23zM1792 1376v-1472q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v1472q0 14 9 23t23 9h192q14 0 23 -9t9 -23z" />
+<glyph unicode="" d="M1024 640q0 106 -75 181t-181 75t-181 -75t-75 -181t75 -181t181 -75t181 75t75 181zM1536 749v-222q0 -12 -8 -23t-20 -13l-185 -28q-19 -54 -39 -91q35 -50 107 -138q10 -12 10 -25t-9 -23q-27 -37 -99 -108t-94 -71q-12 0 -26 9l-138 108q-44 -23 -91 -38 q-16 -136 -29 -186q-7 -28 -36 -28h-222q-14 0 -24.5 8.5t-11.5 21.5l-28 184q-49 16 -90 37l-141 -107q-10 -9 -25 -9q-14 0 -25 11q-126 114 -165 168q-7 10 -7 23q0 12 8 23q15 21 51 66.5t54 70.5q-27 50 -41 99l-183 27q-13 2 -21 12.5t-8 23.5v222q0 12 8 23t19 13 l186 28q14 46 39 92q-40 57 -107 138q-10 12 -10 24q0 10 9 23q26 36 98.5 107.5t94.5 71.5q13 0 26 -10l138 -107q44 23 91 38q16 136 29 186q7 28 36 28h222q14 0 24.5 -8.5t11.5 -21.5l28 -184q49 -16 90 -37l142 107q9 9 24 9q13 0 25 -10q129 -119 165 -170q7 -8 7 -22 q0 -12 -8 -23q-15 -21 -51 -66.5t-54 -70.5q26 -50 41 -98l183 -28q13 -2 21 -12.5t8 -23.5z" />
+<glyph unicode="" horiz-adv-x="1408" d="M512 800v-576q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v576q0 14 9 23t23 9h64q14 0 23 -9t9 -23zM768 800v-576q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v576q0 14 9 23t23 9h64q14 0 23 -9t9 -23zM1024 800v-576q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v576 q0 14 9 23t23 9h64q14 0 23 -9t9 -23zM1152 76v948h-896v-948q0 -22 7 -40.5t14.5 -27t10.5 -8.5h832q3 0 10.5 8.5t14.5 27t7 40.5zM480 1152h448l-48 117q-7 9 -17 11h-317q-10 -2 -17 -11zM1408 1120v-64q0 -14 -9 -23t-23 -9h-96v-948q0 -83 -47 -143.5t-113 -60.5h-832 q-66 0 -113 58.5t-47 141.5v952h-96q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h309l70 167q15 37 54 63t79 26h320q40 0 79 -26t54 -63l70 -167h309q14 0 23 -9t9 -23z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1408 544v-480q0 -26 -19 -45t-45 -19h-384v384h-256v-384h-384q-26 0 -45 19t-19 45v480q0 1 0.5 3t0.5 3l575 474l575 -474q1 -2 1 -6zM1631 613l-62 -74q-8 -9 -21 -11h-3q-13 0 -21 7l-692 577l-692 -577q-12 -8 -24 -7q-13 2 -21 11l-62 74q-8 10 -7 23.5t11 21.5 l719 599q32 26 76 26t76 -26l244 -204v195q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-408l219 -182q10 -8 11 -21.5t-7 -23.5z" />
+<glyph unicode="" d="M1468 1156q28 -28 48 -76t20 -88v-1152q0 -40 -28 -68t-68 -28h-1344q-40 0 -68 28t-28 68v1600q0 40 28 68t68 28h896q40 0 88 -20t76 -48zM1024 1400v-376h376q-10 29 -22 41l-313 313q-12 12 -41 22zM1408 -128v1024h-416q-40 0 -68 28t-28 68v416h-768v-1536h1280z " />
+<glyph unicode="" d="M896 992v-448q0 -14 -9 -23t-23 -9h-320q-14 0 -23 9t-9 23v64q0 14 9 23t23 9h224v352q0 14 9 23t23 9h64q14 0 23 -9t9 -23zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" horiz-adv-x="1920" d="M1111 540v4l-24 320q-1 13 -11 22.5t-23 9.5h-186q-13 0 -23 -9.5t-11 -22.5l-24 -320v-4q-1 -12 8 -20t21 -8h244q12 0 21 8t8 20zM1870 73q0 -73 -46 -73h-704q13 0 22 9.5t8 22.5l-20 256q-1 13 -11 22.5t-23 9.5h-272q-13 0 -23 -9.5t-11 -22.5l-20 -256 q-1 -13 8 -22.5t22 -9.5h-704q-46 0 -46 73q0 54 26 116l417 1044q8 19 26 33t38 14h339q-13 0 -23 -9.5t-11 -22.5l-15 -192q-1 -14 8 -23t22 -9h166q13 0 22 9t8 23l-15 192q-1 13 -11 22.5t-23 9.5h339q20 0 38 -14t26 -33l417 -1044q26 -62 26 -116z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1280 192q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1536 192q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1664 416v-320q0 -40 -28 -68t-68 -28h-1472q-40 0 -68 28t-28 68v320q0 40 28 68t68 28h465l135 -136 q58 -56 136 -56t136 56l136 136h464q40 0 68 -28t28 -68zM1339 985q17 -41 -14 -70l-448 -448q-18 -19 -45 -19t-45 19l-448 448q-31 29 -14 70q17 39 59 39h256v448q0 26 19 45t45 19h256q26 0 45 -19t19 -45v-448h256q42 0 59 -39z" />
+<glyph unicode="" d="M1120 608q0 -12 -10 -24l-319 -319q-11 -9 -23 -9t-23 9l-320 320q-15 16 -7 35q8 20 30 20h192v352q0 14 9 23t23 9h192q14 0 23 -9t9 -23v-352h192q14 0 23 -9t9 -23zM768 1184q-148 0 -273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273 t-73 273t-198 198t-273 73zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" d="M1118 660q-8 -20 -30 -20h-192v-352q0 -14 -9 -23t-23 -9h-192q-14 0 -23 9t-9 23v352h-192q-14 0 -23 9t-9 23q0 12 10 24l319 319q11 9 23 9t23 -9l320 -320q15 -16 7 -35zM768 1184q-148 0 -273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198 t73 273t-73 273t-198 198t-273 73zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" d="M1023 576h316q-1 3 -2.5 8t-2.5 8l-212 496h-708l-212 -496q-1 -2 -2.5 -8t-2.5 -8h316l95 -192h320zM1536 546v-482q0 -26 -19 -45t-45 -19h-1408q-26 0 -45 19t-19 45v482q0 62 25 123l238 552q10 25 36.5 42t52.5 17h832q26 0 52.5 -17t36.5 -42l238 -552 q25 -61 25 -123z" />
+<glyph unicode="" d="M1184 640q0 -37 -32 -55l-544 -320q-15 -9 -32 -9q-16 0 -32 8q-32 19 -32 56v640q0 37 32 56q33 18 64 -1l544 -320q32 -18 32 -55zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" d="M1536 1280v-448q0 -26 -19 -45t-45 -19h-448q-42 0 -59 40q-17 39 14 69l138 138q-148 137 -349 137q-104 0 -198.5 -40.5t-163.5 -109.5t-109.5 -163.5t-40.5 -198.5t40.5 -198.5t109.5 -163.5t163.5 -109.5t198.5 -40.5q119 0 225 52t179 147q7 10 23 12q14 0 25 -9 l137 -138q9 -8 9.5 -20.5t-7.5 -22.5q-109 -132 -264 -204.5t-327 -72.5q-156 0 -298 61t-245 164t-164 245t-61 298t61 298t164 245t245 164t298 61q147 0 284.5 -55.5t244.5 -156.5l130 129q29 31 70 14q39 -17 39 -59z" />
+<glyph unicode="" d="M1511 480q0 -5 -1 -7q-64 -268 -268 -434.5t-478 -166.5q-146 0 -282.5 55t-243.5 157l-129 -129q-19 -19 -45 -19t-45 19t-19 45v448q0 26 19 45t45 19h448q26 0 45 -19t19 -45t-19 -45l-137 -137q71 -66 161 -102t187 -36q134 0 250 65t186 179q11 17 53 117 q8 23 30 23h192q13 0 22.5 -9.5t9.5 -22.5zM1536 1280v-448q0 -26 -19 -45t-45 -19h-448q-26 0 -45 19t-19 45t19 45l138 138q-148 137 -349 137q-134 0 -250 -65t-186 -179q-11 -17 -53 -117q-8 -23 -30 -23h-199q-13 0 -22.5 9.5t-9.5 22.5v7q65 268 270 434.5t480 166.5 q146 0 284 -55.5t245 -156.5l130 129q19 19 45 19t45 -19t19 -45z" />
+<glyph unicode="" horiz-adv-x="1792" d="M384 352v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM384 608v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5z M384 864v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h64q13 0 22.5 -9.5t9.5 -22.5zM1536 352v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-960q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h960q13 0 22.5 -9.5t9.5 -22.5z M1536 608v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-960q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h960q13 0 22.5 -9.5t9.5 -22.5zM1536 864v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-960q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h960q13 0 22.5 -9.5 t9.5 -22.5zM1664 160v832q0 13 -9.5 22.5t-22.5 9.5h-1472q-13 0 -22.5 -9.5t-9.5 -22.5v-832q0 -13 9.5 -22.5t22.5 -9.5h1472q13 0 22.5 9.5t9.5 22.5zM1792 1248v-1088q0 -66 -47 -113t-113 -47h-1472q-66 0 -1
13 47t-47 113v1088q0 66 47 113t113 47h1472q66 0 113 -47 t47 -113z" />
+<glyph unicode="" horiz-adv-x="1152" d="M320 768h512v192q0 106 -75 181t-181 75t-181 -75t-75 -181v-192zM1152 672v-576q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v576q0 40 28 68t68 28h32v192q0 184 132 316t316 132t316 -132t132 -316v-192h32q40 0 68 -28t28 -68z" />
+<glyph unicode="" horiz-adv-x="1792" d="M320 1280q0 -72 -64 -110v-1266q0 -13 -9.5 -22.5t-22.5 -9.5h-64q-13 0 -22.5 9.5t-9.5 22.5v1266q-64 38 -64 110q0 53 37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1792 1216v-763q0 -25 -12.5 -38.5t-39.5 -27.5q-215 -116 -369 -116q-61 0 -123.5 22t-108.5 48 t-115.5 48t-142.5 22q-192 0 -464 -146q-17 -9 -33 -9q-26 0 -45 19t-19 45v742q0 32 31 55q21 14 79 43q236 120 421 120q107 0 200 -29t219 -88q38 -19 88 -19q54 0 117.5 21t110 47t88 47t54.5 21q26 0 45 -19t19 -45z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1664 650q0 -166 -60 -314l-20 -49l-185 -33q-22 -83 -90.5 -136.5t-156.5 -53.5v-32q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v576q0 14 9 23t23 9h64q14 0 23 -9t9 -23v-32q71 0 130 -35.5t93 -95.5l68 12q29 95 29 193q0 148 -88 279t-236.5 209t-315.5 78 t-315.5 -78t-236.5 -209t-88 -279q0 -98 29 -193l68 -12q34 60 93 95.5t130 35.5v32q0 14 9 23t23 9h64q14 0 23 -9t9 -23v-576q0 -14 -9 -23t-23 -9h-64q-14 0 -23 9t-9 23v32q-88 0 -156.5 53.5t-90.5 136.5l-185 33l-20 49q-60 148 -60 314q0 151 67 291t179 242.5 t266 163.5t320 61t320 -61t266 -163.5t179 -242.5t67 -291z" />
+<glyph unicode="" horiz-adv-x="768" d="M768 1184v-1088q0 -26 -19 -45t-45 -19t-45 19l-333 333h-262q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h262l333 333q19 19 45 19t45 -19t19 -45z" />
+<glyph unicode="" horiz-adv-x="1152" d="M768 1184v-1088q0 -26 -19 -45t-45 -19t-45 19l-333 333h-262q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h262l333 333q19 19 45 19t45 -19t19 -45zM1152 640q0 -76 -42.5 -141.5t-112.5 -93.5q-10 -5 -25 -5q-26 0 -45 18.5t-19 45.5q0 21 12 35.5t29 25t34 23t29 35.5 t12 57t-12 57t-29 35.5t-34 23t-29 25t-12 35.5q0 27 19 45.5t45 18.5q15 0 25 -5q70 -27 112.5 -93t42.5 -142z" />
+<glyph unicode="" horiz-adv-x="1664" d="M768 1184v-1088q0 -26 -19 -45t-45 -19t-45 19l-333 333h-262q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h262l333 333q19 19 45 19t45 -19t19 -45zM1152 640q0 -76 -42.5 -141.5t-112.5 -93.5q-10 -5 -25 -5q-26 0 -45 18.5t-19 45.5q0 21 12 35.5t29 25t34 23t29 35.5 t12 57t-12 57t-29 35.5t-34 23t-29 25t-12 35.5q0 27 19 45.5t45 18.5q15 0 25 -5q70 -27 112.5 -93t42.5 -142zM1408 640q0 -153 -85 -282.5t-225 -188.5q-13 -5 -25 -5q-27 0 -46 19t-19 45q0 39 39 59q56 29 76 44q74 54 115.5 135.5t41.5 173.5t-41.5 173.5 t-115.5 135.5q-20 15 -76 44q-39 20 -39 59q0 26 19 45t45 19q13 0 26 -5q140 -59 225 -188.5t85 -282.5zM1664 640q0 -230 -127 -422.5t-338 -283.5q-13 -5 -26 -5q-26 0 -45 19t-19 45q0 36 39 59q7 4 22.5 10.5t22.5 10.5q46 25 82 51q123 91 192 227t69 289t-69 289 t-192 227q-36 26 -82 51q-7 4 -22.5 10.5t-22.5 10.5q-39 23 -39 59q0 26 19 45t45 19q13 0 26 -5q211 -91 338 -283.5t127 -422.5z" />
+<glyph unicode="" horiz-adv-x="1408" d="M384 384v-128h-128v128h128zM384 1152v-128h-128v128h128zM1152 1152v-128h-128v128h128zM128 129h384v383h-384v-383zM128 896h384v384h-384v-384zM896 896h384v384h-384v-384zM640 640v-640h-640v640h640zM1152 128v-128h-128v128h128zM1408 128v-128h-128v128h128z M1408 640v-384h-384v128h-128v-384h-128v640h384v-128h128v128h128zM640 1408v-640h-640v640h640zM1408 1408v-640h-640v640h640z" />
+<glyph unicode="" horiz-adv-x="1792" d="M63 0h-63v1408h63v-1408zM126 1h-32v1407h32v-1407zM220 1h-31v1407h31v-1407zM377 1h-31v1407h31v-1407zM534 1h-62v1407h62v-1407zM660 1h-31v1407h31v-1407zM723 1h-31v1407h31v-1407zM786 1h-31v1407h31v-1407zM943 1h-63v1407h63v-1407zM1100 1h-63v1407h63v-1407z M1226 1h-63v1407h63v-1407zM1352 1h-63v1407h63v-1407zM1446 1h-63v1407h63v-1407zM1635 1h-94v1407h94v-1407zM1698 1h-32v1407h32v-1407zM1792 0h-63v1408h63v-1408z" />
+<glyph unicode="" d="M448 1088q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1515 512q0 -53 -37 -90l-491 -492q-39 -37 -91 -37q-53 0 -90 37l-715 716q-38 37 -64.5 101t-26.5 117v416q0 52 38 90t90 38h416q53 0 117 -26.5t102 -64.5 l715 -714q37 -39 37 -91z" />
+<glyph unicode="" horiz-adv-x="1920" d="M448 1088q0 53 -37.5 90.5t-90.5 37.5t-90.5 -37.5t-37.5 -90.5t37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1515 512q0 -53 -37 -90l-491 -492q-39 -37 -91 -37q-53 0 -90 37l-715 716q-38 37 -64.5 101t-26.5 117v416q0 52 38 90t90 38h416q53 0 117 -26.5t102 -64.5 l715 -714q37 -39 37 -91zM1899 512q0 -53 -37 -90l-491 -492q-39 -37 -91 -37q-36 0 -59 14t-53 45l470 470q37 37 37 90q0 52 -37 91l-715 714q-38 38 -102 64.5t-117 26.5h224q53 0 117 -26.5t102 -64.5l715 -714q37 -39 37 -91z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1639 1058q40 -57 18 -129l-275 -906q-19 -64 -76.5 -107.5t-122.5 -43.5h-923q-77 0 -148.5 53.5t-99.5 131.5q-24 67 -2 127q0 4 3 27t4 37q1 8 -3 21.5t-3 19.5q2 11 8 21t16.5 23.5t16.5 23.5q23 38 45 91.5t30 91.5q3 10 0.5 30t-0.5 28q3 11 17 28t17 23 q21 36 42 92t25 90q1 9 -2.5 32t0.5 28q4 13 22 30.5t22 22.5q19 26 42.5 84.5t27.5 96.5q1 8 -3 25.5t-2 26.5q2 8 9 18t18 23t17 21q8 12 16.5 30.5t15 35t16 36t19.5 32t26.5 23.5t36 11.5t47.5 -5.5l-1 -3q38 9 51 9h761q74 0 114 -56t18 -130l-274 -906 q-36 -119 -71.5 -153.5t-128.5 -34.5h-869q-27 0 -38 -15q-11 -16 -1 -43q24 -70 144 -70h923q29 0 56 15.5t35 41.5l300 987q7 22 5 57q38 -15 59 -43zM575 1056q-4 -13 2 -22.5t20 -9.5h608q13 0 25.5 9.5t16.5 22.5l21 64q4 13 -2 22.5t-20 9.5h-608q-13 0 -25.5 -9.5 t-16.5 -22.5zM492 800q-4 -13 2 -22.5t20 -9.5h608q13 0 25.5 9.5t16.5 22.5l21 64q4 13 -2 22.5t-20 9.5h-608q-13 0 -25.5 -9.5t-16.5 -22.5z" />
+<glyph unicode="" horiz-adv-x="1280" d="M1164 1408q23 0 44 -9q33 -13 52.5 -41t19.5 -62v-1289q0 -34 -19.5 -62t-52.5 -41q-19 -8 -44 -8q-48 0 -83 32l-441 424l-441 -424q-36 -33 -83 -33q-23 0 -44 9q-33 13 -52.5 41t-19.5 62v1289q0 34 19.5 62t52.5 41q21 9 44 9h1048z" />
+<glyph unicode="" horiz-adv-x="1664" d="M384 0h896v256h-896v-256zM384 640h896v384h-160q-40 0 -68 28t-28 68v160h-640v-640zM1536 576q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1664 576v-416q0 -13 -9.5 -22.5t-22.5 -9.5h-224v-160q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68 v160h-224q-13 0 -22.5 9.5t-9.5 22.5v416q0 79 56.5 135.5t135.5 56.5h64v544q0 40 28 68t68 28h672q40 0 88 -20t76 -48l152 -152q28 -28 48 -76t20 -88v-256h64q79 0 135.5 -56.5t56.5 -135.5z" />
+<glyph unicode="" horiz-adv-x="1920" d="M960 864q119 0 203.5 -84.5t84.5 -203.5t-84.5 -203.5t-203.5 -84.5t-203.5 84.5t-84.5 203.5t84.5 203.5t203.5 84.5zM1664 1280q106 0 181 -75t75 -181v-896q0 -106 -75 -181t-181 -75h-1408q-106 0 -181 75t-75 181v896q0 106 75 181t181 75h224l51 136 q19 49 69.5 84.5t103.5 35.5h512q53 0 103.5 -35.5t69.5 -84.5l51 -136h224zM960 128q185 0 316.5 131.5t131.5 316.5t-131.5 316.5t-316.5 131.5t-316.5 -131.5t-131.5 -316.5t131.5 -316.5t316.5 -131.5z" />
+<glyph unicode="" horiz-adv-x="1664" d="M725 977l-170 -450q33 0 136.5 -2t160.5 -2q19 0 57 2q-87 253 -184 452zM0 -128l2 79q23 7 56 12.5t57 10.5t49.5 14.5t44.5 29t31 50.5l237 616l280 724h75h53q8 -14 11 -21l205 -480q33 -78 106 -257.5t114 -274.5q15 -34 58 -144.5t72 -168.5q20 -45 35 -57 q19 -15 88 -29.5t84 -20.5q6 -38 6 -57q0 -4 -0.5 -13t-0.5 -13q-63 0 -190 8t-191 8q-76 0 -215 -7t-178 -8q0 43 4 78l131 28q1 0 12.5 2.5t15.5 3.5t14.5 4.5t15 6.5t11 8t9 11t2.5 14q0 16 -31 96.5t-72 177.5t-42 100l-450 2q-26 -58 -76.5 -195.5t-50.5 -162.5 q0 -22 14 -37.5t43.5 -24.5t48.5 -13.5t57 -8.5t41 -4q1 -19 1 -58q0 -9 -2 -27q-58 0 -174.5 10t-174.5 10q-8 0 -26.5 -4t-21.5 -4q-80 -14 -188 -14z" />
+<glyph unicode="" horiz-adv-x="1408" d="M555 15q74 -32 140 -32q376 0 376 335q0 114 -41 180q-27 44 -61.5 74t-67.5 46.5t-80.5 25t-84 10.5t-94.5 2q-73 0 -101 -10q0 -53 -0.5 -159t-0.5 -158q0 -8 -1 -67.5t-0.5 -96.5t4.5 -83.5t12 -66.5zM541 761q42 -7 109 -7q82 0 143 13t110 44.5t74.5 89.5t25.5 142 q0 70 -29 122.5t-79 82t-108 43.5t-124 14q-50 0 -130 -13q0 -50 4 -151t4 -152q0 -27 -0.5 -80t-0.5 -79q0 -46 1 -69zM0 -128l2 94q15 4 85 16t106 27q7 12 12.5 27t8.5 33.5t5.5 32.5t3 37.5t0.5 34v35.5v30q0 982 -22 1025q-4 8 -22 14.5t-44.5 11t-49.5 7t-48.5 4.5 t-30.5 3l-4 83q98 2 340 11.5t373 9.5q23 0 68.5 -0.5t67.5 -0.5q70 0 136.5 -13t128.5 -42t108 -71t74 -104.5t28 -137.5q0 -52 -16.5 -95.5t-39 -72t-64.5 -57.5t-73 -45t-84 -40q154 -35 256.5 -134t102.5 -248q0 -100 -35 -179.5t-93.5 -130.5t-138 -85.5t-163.5 -48.5 t-176 -14q-44 0 -132 3t-132 3q-106 0 -307 -11t-231 -12z" />
+<glyph unicode="" horiz-adv-x="1024" d="M0 -126l17 85q6 2 81.5 21.5t111.5 37.5q28 35 41 101q1 7 62 289t114 543.5t52 296.5v25q-24 13 -54.5 18.5t-69.5 8t-58 5.5l19 103q33 -2 120 -6.5t149.5 -7t120.5 -2.5q48 0 98.5 2.5t121 7t98.5 6.5q-5 -39 -19 -89q-30 -10 -101.5 -28.5t-108.5 -33.5 q-8 -19 -14 -42.5t-9 -40t-7.5 -45.5t-6.5 -42q-27 -148 -87.5 -419.5t-77.5 -355.5q-2 -9 -13 -58t-20 -90t-16 -83.5t-6 -57.5l1 -18q17 -4 185 -31q-3 -44 -16 -99q-11 0 -32.5 -1.5t-32.5 -1.5q-29 0 -87 10t-86 10q-138 2 -206 2q-51 0 -143 -9t-121 -11z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1744 128q33 0 42 -18.5t-11 -44.5l-126 -162q-20 -26 -49 -26t-49 26l-126 162q-20 26 -11 44.5t42 18.5h80v1024h-80q-33 0 -42 18.5t11 44.5l126 162q20 26 49 26t49 -26l126 -162q20 -26 11 -44.5t-42 -18.5h-80v-1024h80zM81 1407l54 -27q12 -5 211 -5q44 0 132 2 t132 2q36 0 107.5 -0.5t107.5 -0.5h293q6 0 21 -0.5t20.5 0t16 3t17.5 9t15 17.5l42 1q4 0 14 -0.5t14 -0.5q2 -112 2 -336q0 -80 -5 -109q-39 -14 -68 -18q-25 44 -54 128q-3 9 -11 48t-14.5 73.5t-7.5 35.5q-6 8 -12 12.5t-15.5 6t-13 2.5t-18 0.5t-16.5 -0.5 q-17 0 -66.5 0.5t-74.5 0.5t-64 -2t-71 -6q-9 -81 -8 -136q0 -94 2 -388t2 -455q0 -16 -2.5 -71.5t0 -91.5t12.5 -69q40 -21 124 -42.5t120 -37.5q5 -40 5 -50q0 -14 -3 -29l-34 -1q-76 -2 -218 8t-207 10q-50 0 -151 -9t-152 -9q-3 51 -3 52v9q17 27 61.5 43t98.5 29t78 27 q19 42 19 383q0 101 -3 303t-3 303v117q0 2 0.5 15.5t0.5 25t-1 25.5t-3 24t-5 14q-11 12 -162 12q-33 0 -93 -12t-80 -26q-19 -13 -34 -72.5t-31.5 -111t-42.5 -53.5q-42 26 -56 44v383z" />
+<glyph unicode="" d="M81 1407l54 -27q12 -5 211 -5q44 0 132 2t132 2q70 0 246.5 1t304.5 0.5t247 -4.5q33 -1 56 31l42 1q4 0 14 -0.5t14 -0.5q2 -112 2 -336q0 -80 -5 -109q-39 -14 -68 -18q-25 44 -54 128q-3 9 -11 47.5t-15 73.5t-7 36q-10 13 -27 19q-5 2 -66 2q-30 0 -93 1t-103 1 t-94 -2t-96 -7q-9 -81 -8 -136l1 -152v52q0 -55 1 -154t1.5 -180t0.5 -153q0 -16 -2.5 -71.5t0 -91.5t12.5 -69q40 -21 124 -42.5t120 -37.5q5 -40 5 -50q0 -14 -3 -29l-34 -1q-76 -2 -218 8t-207 10q-50 0 -151 -9t-152 -9q-3 51 -3 52v9q17 27 61.5 43t98.5 29t78 27 q7 16 11.5 74t6 145.5t1.5 155t-0.5 153.5t-0.5 89q0 7 -2.5 21.5t-2.5 22.5q0 7 0.5 44t1 73t0 76.5t-3 67.5t-6.5 32q-11 12 -162 12q-41 0 -163 -13.5t-138 -24.5q-19 -12 -34 -71.5t-31.5 -111.5t-42.5 -54q-42 26 -56 44v383zM1310 125q12 0 42 -19.5t57.5 -41.5 t59.5 -49t36 -30q26 -21 26 -49t-26 -49q-4 -3 -36 -30t-59.5 -49t-57.5 -41.5t-42 -19.5q-13 0 -20.5 10.5t-10 28.5t-2.5 33.5t1.5 33t1.5 19.5h-1024q0 -2 1.5 -19.5t1.5 -33t-2.5 -33.5t-10 -28.5t-20.5 -10.5q-12 0 -42 19.5t-57.5 41
.5t-59.5 49t-36 30q-26 21 -26 49 t26 49q4 3 36 30t59.5 49t57.5 41.5t42 19.5q13 0 20.5 -10.5t10 -28.5t2.5 -33.5t-1.5 -33t-1.5 -19.5h1024q0 2 -1.5 19.5t-1.5 33t2.5 33.5t10 28.5t20.5 10.5z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1792 192v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1408 576v-128q0 -26 -19 -45t-45 -19h-1280q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1280q26 0 45 -19t19 -45zM1664 960v-128q0 -26 -19 -45 t-45 -19h-1536q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1536q26 0 45 -19t19 -45zM1280 1344v-128q0 -26 -19 -45t-45 -19h-1152q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1152q26 0 45 -19t19 -45z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1792 192v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1408 576v-128q0 -26 -19 -45t-45 -19h-896q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h896q26 0 45 -19t19 -45zM1664 960v-128q0 -26 -19 -45t-45 -19 h-1408q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1408q26 0 45 -19t19 -45zM1280 1344v-128q0 -26 -19 -45t-45 -19h-640q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h640q26 0 45 -19t19 -45z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1792 192v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 576v-128q0 -26 -19 -45t-45 -19h-1280q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1280q26 0 45 -19t19 -45zM1792 960v-128q0 -26 -19 -45 t-45 -19h-1536q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1536q26 0 45 -19t19 -45zM1792 1344v-128q0 -26 -19 -45t-45 -19h-1152q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1152q26 0 45 -19t19 -45z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1792 192v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 576v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 960v-128q0 -26 -19 -45 t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45zM1792 1344v-128q0 -26 -19 -45t-45 -19h-1664q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h1664q26 0 45 -19t19 -45z" />
+<glyph unicode="" horiz-adv-x="1792" d="M256 224v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-192q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h192q13 0 22.5 -9.5t9.5 -22.5zM256 608v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-192q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h192q13 0 22.5 -9.5 t9.5 -22.5zM256 992v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-192q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h192q13 0 22.5 -9.5t9.5 -22.5zM1792 224v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1344q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1344 q13 0 22.5 -9.5t9.5 -22.5zM256 1376v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-192q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h192q13 0 22.5 -9.5t9.5 -22.5zM1792 608v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1344q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5 t22.5 9.5h1344q13 0 22.5 -9.5t9.5 -22.5zM1792 992v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1344q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1344q13 0 22.5 -9.5t9.5 -22.5zM1792 1376v-192q0 -13 -9.5 -22.5t
-22.5 -9.5h-1344q-13 0 -22.5 9.5t-9.5 22.5v192 q0 13 9.5 22.5t22.5 9.5h1344q13 0 22.5 -9.5t9.5 -22.5z" />
+<glyph unicode="" horiz-adv-x="1792" d="M384 992v-576q0 -13 -9.5 -22.5t-22.5 -9.5q-14 0 -23 9l-288 288q-9 9 -9 23t9 23l288 288q9 9 23 9q13 0 22.5 -9.5t9.5 -22.5zM1792 224v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1728q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1728q13 0 22.5 -9.5 t9.5 -22.5zM1792 608v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1088q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1088q13 0 22.5 -9.5t9.5 -22.5zM1792 992v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1088q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1088 q13 0 22.5 -9.5t9.5 -22.5zM1792 1376v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1728q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1728q13 0 22.5 -9.5t9.5 -22.5z" />
+<glyph unicode="" horiz-adv-x="1792" d="M352 704q0 -14 -9 -23l-288 -288q-9 -9 -23 -9q-13 0 -22.5 9.5t-9.5 22.5v576q0 13 9.5 22.5t22.5 9.5q14 0 23 -9l288 -288q9 -9 9 -23zM1792 224v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1728q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1728q13 0 22.5 -9.5 t9.5 -22.5zM1792 608v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1088q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1088q13 0 22.5 -9.5t9.5 -22.5zM1792 992v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1088q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1088 q13 0 22.5 -9.5t9.5 -22.5zM1792 1376v-192q0 -13 -9.5 -22.5t-22.5 -9.5h-1728q-13 0 -22.5 9.5t-9.5 22.5v192q0 13 9.5 22.5t22.5 9.5h1728q13 0 22.5 -9.5t9.5 -22.5z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1792 1184v-1088q0 -42 -39 -59q-13 -5 -25 -5q-27 0 -45 19l-403 403v-166q0 -119 -84.5 -203.5t-203.5 -84.5h-704q-119 0 -203.5 84.5t-84.5 203.5v704q0 119 84.5 203.5t203.5 84.5h704q119 0 203.5 -84.5t84.5 -203.5v-165l403 402q18 19 45 19q12 0 25 -5 q39 -17 39 -59z" />
+<glyph unicode="" horiz-adv-x="1920" d="M640 960q0 -80 -56 -136t-136 -56t-136 56t-56 136t56 136t136 56t136 -56t56 -136zM1664 576v-448h-1408v192l320 320l160 -160l512 512zM1760 1280h-1600q-13 0 -22.5 -9.5t-9.5 -22.5v-1216q0 -13 9.5 -22.5t22.5 -9.5h1600q13 0 22.5 9.5t9.5 22.5v1216 q0 13 -9.5 22.5t-22.5 9.5zM1920 1248v-1216q0 -66 -47 -113t-113 -47h-1600q-66 0 -113 47t-47 113v1216q0 66 47 113t113 47h1600q66 0 113 -47t47 -113z" />
+<glyph unicode="" d="M363 0l91 91l-235 235l-91 -91v-107h128v-128h107zM886 928q0 22 -22 22q-10 0 -17 -7l-542 -542q-7 -7 -7 -17q0 -22 22 -22q10 0 17 7l542 542q7 7 7 17zM832 1120l416 -416l-832 -832h-416v416zM1515 1024q0 -53 -37 -90l-166 -166l-416 416l166 165q36 38 90 38 q53 0 91 -38l235 -234q37 -39 37 -91z" />
+<glyph unicode="" horiz-adv-x="1024" d="M768 896q0 106 -75 181t-181 75t-181 -75t-75 -181t75 -181t181 -75t181 75t75 181zM1024 896q0 -109 -33 -179l-364 -774q-16 -33 -47.5 -52t-67.5 -19t-67.5 19t-46.5 52l-365 774q-33 70 -33 179q0 212 150 362t362 150t362 -150t150 -362z" />
+<glyph unicode="" d="M768 96v1088q-148 0 -273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" horiz-adv-x="1024" d="M512 384q0 36 -20 69q-1 1 -15.5 22.5t-25.5 38t-25 44t-21 50.5q-4 16 -21 16t-21 -16q-7 -23 -21 -50.5t-25 -44t-25.5 -38t-15.5 -22.5q-20 -33 -20 -69q0 -53 37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1024 512q0 -212 -150 -362t-362 -150t-362 150t-150 362 q0 145 81 275q6 9 62.5 90.5t101 151t99.5 178t83 201.5q9 30 34 47t51 17t51.5 -17t33.5 -47q28 -93 83 -201.5t99.5 -178t101 -151t62.5 -90.5q81 -127 81 -275z" />
+<glyph unicode="" horiz-adv-x="1792" d="M888 352l116 116l-152 152l-116 -116v-56h96v-96h56zM1328 1072q-16 16 -33 -1l-350 -350q-17 -17 -1 -33t33 1l350 350q17 17 1 33zM1408 478v-190q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h832 q63 0 117 -25q15 -7 18 -23q3 -17 -9 -29l-49 -49q-14 -14 -32 -8q-23 6 -45 6h-832q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832q66 0 113 47t47 113v126q0 13 9 22l64 64q15 15 35 7t20 -29zM1312 1216l288 -288l-672 -672h-288v288zM1756 1084l-92 -92 l-288 288l92 92q28 28 68 28t68 -28l152 -152q28 -28 28 -68t-28 -68z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1408 547v-259q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h255v0q13 0 22.5 -9.5t9.5 -22.5q0 -27 -26 -32q-77 -26 -133 -60q-10 -4 -16 -4h-112q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832 q66 0 113 47t47 113v214q0 19 18 29q28 13 54 37q16 16 35 8q21 -9 21 -29zM1645 1043l-384 -384q-18 -19 -45 -19q-12 0 -25 5q-39 17 -39 59v192h-160q-323 0 -438 -131q-119 -137 -74 -473q3 -23 -20 -34q-8 -2 -12 -2q-16 0 -26 13q-10 14 -21 31t-39.5 68.5t-49.5 99.5 t-38.5 114t-17.5 122q0 49 3.5 91t14 90t28 88t47 81.5t68.5 74t94.5 61.5t124.5 48.5t159.5 30.5t196.5 11h160v192q0 42 39 59q13 5 25 5q26 0 45 -19l384 -384q19 -19 19 -45t-19 -45z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1408 606v-318q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h832q63 0 117 -25q15 -7 18 -23q3 -17 -9 -29l-49 -49q-10 -10 -23 -10q-3 0 -9 2q-23 6 -45 6h-832q-66 0 -113 -47t-47 -113v-832 q0 -66 47 -113t113 -47h832q66 0 113 47t47 113v254q0 13 9 22l64 64q10 10 23 10q6 0 12 -3q20 -8 20 -29zM1639 1095l-814 -814q-24 -24 -57 -24t-57 24l-430 430q-24 24 -24 57t24 57l110 110q24 24 57 24t57 -24l263 -263l647 647q24 24 57 24t57 -24l110 -110 q24 -24 24 -57t-24 -57z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1792 640q0 -26 -19 -45l-256 -256q-19 -19 -45 -19t-45 19t-19 45v128h-384v-384h128q26 0 45 -19t19 -45t-19 -45l-256 -256q-19 -19 -45 -19t-45 19l-256 256q-19 19 -19 45t19 45t45 19h128v384h-384v-128q0 -26 -19 -45t-45 -19t-45 19l-256 256q-19 19 -19 45 t19 45l256 256q19 19 45 19t45 -19t19 -45v-128h384v384h-128q-26 0 -45 19t-19 45t19 45l256 256q19 19 45 19t45 -19l256 -256q19 -19 19 -45t-19 -45t-45 -19h-128v-384h384v128q0 26 19 45t45 19t45 -19l256 -256q19 -19 19 -45z" />
+<glyph unicode="" horiz-adv-x="1024" d="M979 1395q19 19 32 13t13 -32v-1472q0 -26 -13 -32t-32 13l-710 710q-9 9 -13 19v-678q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-678q4 11 13 19z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1747 1395q19 19 32 13t13 -32v-1472q0 -26 -13 -32t-32 13l-710 710q-9 9 -13 19v-710q0 -26 -13 -32t-32 13l-710 710q-9 9 -13 19v-678q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-678q4 11 13 19l710 710 q19 19 32 13t13 -32v-710q4 11 13 19z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1619 1395q19 19 32 13t13 -32v-1472q0 -26 -13 -32t-32 13l-710 710q-8 9 -13 19v-710q0 -26 -13 -32t-32 13l-710 710q-19 19 -19 45t19 45l710 710q19 19 32 13t13 -32v-710q5 11 13 19z" />
+<glyph unicode="" horiz-adv-x="1408" d="M1384 609l-1328 -738q-23 -13 -39.5 -3t-16.5 36v1472q0 26 16.5 36t39.5 -3l1328 -738q23 -13 23 -31t-23 -31z" />
+<glyph unicode="" d="M1536 1344v-1408q0 -26 -19 -45t-45 -19h-512q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h512q26 0 45 -19t19 -45zM640 1344v-1408q0 -26 -19 -45t-45 -19h-512q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h512q26 0 45 -19t19 -45z" />
+<glyph unicode="" d="M1536 1344v-1408q0 -26 -19 -45t-45 -19h-1408q-26 0 -45 19t-19 45v1408q0 26 19 45t45 19h1408q26 0 45 -19t19 -45z" />
+<glyph unicode="" horiz-adv-x="1664" d="M45 -115q-19 -19 -32 -13t-13 32v1472q0 26 13 32t32 -13l710 -710q8 -8 13 -19v710q0 26 13 32t32 -13l710 -710q19 -19 19 -45t-19 -45l-710 -710q-19 -19 -32 -13t-13 32v710q-5 -10 -13 -19z" />
+<glyph unicode="" horiz-adv-x="1792" d="M45 -115q-19 -19 -32 -13t-13 32v1472q0 26 13 32t32 -13l710 -710q8 -8 13 -19v710q0 26 13 32t32 -13l710 -710q8 -8 13 -19v678q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-1408q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v678q-5 -10 -13 -19l-710 -710 q-19 -19 -32 -13t-13 32v710q-5 -10 -13 -19z" />
+<glyph unicode="" horiz-adv-x="1024" d="M45 -115q-19 -19 -32 -13t-13 32v1472q0 26 13 32t32 -13l710 -710q8 -8 13 -19v678q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-1408q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v678q-5 -10 -13 -19z" />
+<glyph unicode="" horiz-adv-x="1538" d="M14 557l710 710q19 19 45 19t45 -19l710 -710q19 -19 13 -32t-32 -13h-1472q-26 0 -32 13t13 32zM1473 0h-1408q-26 0 -45 19t-19 45v256q0 26 19 45t45 19h1408q26 0 45 -19t19 -45v-256q0 -26 -19 -45t-45 -19z" />
+<glyph unicode="" horiz-adv-x="1280" d="M1171 1235l-531 -531l531 -531q19 -19 19 -45t-19 -45l-166 -166q-19 -19 -45 -19t-45 19l-742 742q-19 19 -19 45t19 45l742 742q19 19 45 19t45 -19l166 -166q19 -19 19 -45t-19 -45z" />
+<glyph unicode="" horiz-adv-x="1280" d="M1107 659l-742 -742q-19 -19 -45 -19t-45 19l-166 166q-19 19 -19 45t19 45l531 531l-531 531q-19 19 -19 45t19 45l166 166q19 19 45 19t45 -19l742 -742q19 -19 19 -45t-19 -45z" />
+<glyph unicode="" d="M1216 576v128q0 26 -19 45t-45 19h-256v256q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-256h-256q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h256v-256q0 -26 19 -45t45 -19h128q26 0 45 19t19 45v256h256q26 0 45 19t19 45zM1536 640q0 -209 -103 -385.5 t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" d="M1216 576v128q0 26 -19 45t-45 19h-768q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h768q26 0 45 19t19 45zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5 t103 -385.5z" />
+<glyph unicode="" d="M1149 414q0 26 -19 45l-181 181l181 181q19 19 19 45q0 27 -19 46l-90 90q-19 19 -46 19q-26 0 -45 -19l-181 -181l-181 181q-19 19 -45 19q-27 0 -46 -19l-90 -90q-19 -19 -19 -46q0 -26 19 -45l181 -181l-181 -181q-19 -19 -19 -45q0 -27 19 -46l90 -90q19 -19 46 -19 q26 0 45 19l181 181l181 -181q19 -19 45 -19q27 0 46 19l90 90q19 19 19 46zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" d="M1284 802q0 28 -18 46l-91 90q-19 19 -45 19t-45 -19l-408 -407l-226 226q-19 19 -45 19t-45 -19l-91 -90q-18 -18 -18 -46q0 -27 18 -45l362 -362q19 -19 45 -19q27 0 46 19l543 543q18 18 18 45zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103 t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" d="M896 160v192q0 14 -9 23t-23 9h-192q-14 0 -23 -9t-9 -23v-192q0 -14 9 -23t23 -9h192q14 0 23 9t9 23zM1152 832q0 88 -55.5 163t-138.5 116t-170 41q-243 0 -371 -213q-15 -24 8 -42l132 -100q7 -6 19 -6q16 0 25 12q53 68 86 92q34 24 86 24q48 0 85.5 -26t37.5 -59 q0 -38 -20 -61t-68 -45q-63 -28 -115.5 -86.5t-52.5 -125.5v-36q0 -14 9 -23t23 -9h192q14 0 23 9t9 23q0 19 21.5 49.5t54.5 49.5q32 18 49 28.5t46 35t44.5 48t28 60.5t12.5 81zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5 t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" d="M1024 160v160q0 14 -9 23t-23 9h-96v512q0 14 -9 23t-23 9h-320q-14 0 -23 -9t-9 -23v-160q0 -14 9 -23t23 -9h96v-320h-96q-14 0 -23 -9t-9 -23v-160q0 -14 9 -23t23 -9h448q14 0 23 9t9 23zM896 1056v160q0 14 -9 23t-23 9h-192q-14 0 -23 -9t-9 -23v-160q0 -14 9 -23 t23 -9h192q14 0 23 9t9 23zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" d="M1197 512h-109q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h109q-32 108 -112.5 188.5t-188.5 112.5v-109q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v109q-108 -32 -188.5 -112.5t-112.5 -188.5h109q26 0 45 -19t19 -45v-128q0 -26 -19 -45t-45 -19h-109 q32 -108 112.5 -188.5t188.5 -112.5v109q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-109q108 32 188.5 112.5t112.5 188.5zM1536 704v-128q0 -26 -19 -45t-45 -19h-143q-37 -161 -154.5 -278.5t-278.5 -154.5v-143q0 -26 -19 -45t-45 -19h-128q-26 0 -45 19t-19 45v143 q-161 37 -278.5 154.5t-154.5 278.5h-143q-26 0 -45 19t-19 45v128q0 26 19 45t45 19h143q37 161 154.5 278.5t278.5 154.5v143q0 26 19 45t45 19h128q26 0 45 -19t19 -45v-143q161 -37 278.5 -154.5t154.5 -278.5h143q26 0 45 -19t19 -45z" />
+<glyph unicode="" d="M1097 457l-146 -146q-10 -10 -23 -10t-23 10l-137 137l-137 -137q-10 -10 -23 -10t-23 10l-146 146q-10 10 -10 23t10 23l137 137l-137 137q-10 10 -10 23t10 23l146 146q10 10 23 10t23 -10l137 -137l137 137q10 10 23 10t23 -10l146 -146q10 -10 10 -23t-10 -23 l-137 -137l137 -137q10 -10 10 -23t-10 -23zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198t-73 -273t73 -273t198 -198t273 -73t273 73t198 198t73 273zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5 t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" d="M1171 723l-422 -422q-19 -19 -45 -19t-45 19l-294 294q-19 19 -19 45t19 45l102 102q19 19 45 19t45 -19l147 -147l275 275q19 19 45 19t45 -19l102 -102q19 -19 19 -45t-19 -45zM1312 640q0 148 -73 273t-198 198t-273 73t-273 -73t-198 -198t-73 -273t73 -273t198 -198 t273 -73t273 73t198 198t73 273zM1536 640q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" d="M1312 643q0 161 -87 295l-754 -753q137 -89 297 -89q111 0 211.5 43.5t173.5 116.5t116 174.5t43 212.5zM313 344l755 754q-135 91 -300 91q-148 0 -273 -73t-198 -199t-73 -274q0 -162 89 -299zM1536 643q0 -157 -61 -300t-163.5 -246t-245 -164t-298.5 -61t-298.5 61 t-245 164t-163.5 246t-61 300t61 299.5t163.5 245.5t245 164t298.5 61t298.5 -61t245 -164t163.5 -245.5t61 -299.5z" />
+<glyph unicode="" d="M1536 640v-128q0 -53 -32.5 -90.5t-84.5 -37.5h-704l293 -294q38 -36 38 -90t-38 -90l-75 -76q-37 -37 -90 -37q-52 0 -91 37l-651 652q-37 37 -37 90q0 52 37 91l651 650q38 38 91 38q52 0 90 -38l75 -74q38 -38 38 -91t-38 -91l-293 -293h704q52 0 84.5 -37.5 t32.5 -90.5z" />
+<glyph unicode="" d="M1472 576q0 -54 -37 -91l-651 -651q-39 -37 -91 -37q-51 0 -90 37l-75 75q-38 38 -38 91t38 91l293 293h-704q-52 0 -84.5 37.5t-32.5 90.5v128q0 53 32.5 90.5t84.5 37.5h704l-293 294q-38 36 -38 90t38 90l75 75q38 38 90 38q53 0 91 -38l651 -651q37 -35 37 -90z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1611 565q0 -51 -37 -90l-75 -75q-38 -38 -91 -38q-54 0 -90 38l-294 293v-704q0 -52 -37.5 -84.5t-90.5 -32.5h-128q-53 0 -90.5 32.5t-37.5 84.5v704l-294 -293q-36 -38 -90 -38t-90 38l-75 75q-38 38 -38 90q0 53 38 91l651 651q35 37 90 37q54 0 91 -37l651 -651 q37 -39 37 -91z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1611 704q0 -53 -37 -90l-651 -652q-39 -37 -91 -37q-53 0 -90 37l-651 652q-38 36 -38 90q0 53 38 91l74 75q39 37 91 37q53 0 90 -37l294 -294v704q0 52 38 90t90 38h128q52 0 90 -38t38 -90v-704l294 294q37 37 90 37q52 0 91 -37l75 -75q37 -39 37 -91z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1792 896q0 -26 -19 -45l-512 -512q-19 -19 -45 -19t-45 19t-19 45v256h-224q-98 0 -175.5 -6t-154 -21.5t-133 -42.5t-105.5 -69.5t-80 -101t-48.5 -138.5t-17.5 -181q0 -55 5 -123q0 -6 2.5 -23.5t2.5 -26.5q0 -15 -8.5 -25t-23.5 -10q-16 0 -28 17q-7 9 -13 22 t-13.5 30t-10.5 24q-127 285 -127 451q0 199 53 333q162 403 875 403h224v256q0 26 19 45t45 19t45 -19l512 -512q19 -19 19 -45z" />
+<glyph unicode="" d="M755 480q0 -13 -10 -23l-332 -332l144 -144q19 -19 19 -45t-19 -45t-45 -19h-448q-26 0 -45 19t-19 45v448q0 26 19 45t45 19t45 -19l144 -144l332 332q10 10 23 10t23 -10l114 -114q10 -10 10 -23zM1536 1344v-448q0 -26 -19 -45t-45 -19t-45 19l-144 144l-332 -332 q-10 -10 -23 -10t-23 10l-114 114q-10 10 -10 23t10 23l332 332l-144 144q-19 19 -19 45t19 45t45 19h448q26 0 45 -19t19 -45z" />
+<glyph unicode="" d="M768 576v-448q0 -26 -19 -45t-45 -19t-45 19l-144 144l-332 -332q-10 -10 -23 -10t-23 10l-114 114q-10 10 -10 23t10 23l332 332l-144 144q-19 19 -19 45t19 45t45 19h448q26 0 45 -19t19 -45zM1523 1248q0 -13 -10 -23l-332 -332l144 -144q19 -19 19 -45t-19 -45 t-45 -19h-448q-26 0 -45 19t-19 45v448q0 26 19 45t45 19t45 -19l144 -144l332 332q10 10 23 10t23 -10l114 -114q10 -10 10 -23z" />
+<glyph unicode="" horiz-adv-x="1408" d="M1408 800v-192q0 -40 -28 -68t-68 -28h-416v-416q0 -40 -28 -68t-68 -28h-192q-40 0 -68 28t-28 68v416h-416q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h416v416q0 40 28 68t68 28h192q40 0 68 -28t28 -68v-416h416q40 0 68 -28t28 -68z" />
+<glyph unicode="" horiz-adv-x="1408" d="M1408 800v-192q0 -40 -28 -68t-68 -28h-1216q-40 0 -68 28t-28 68v192q0 40 28 68t68 28h1216q40 0 68 -28t28 -68z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1482 486q46 -26 59.5 -77.5t-12.5 -97.5l-64 -110q-26 -46 -77.5 -59.5t-97.5 12.5l-266 153v-307q0 -52 -38 -90t-90 -38h-128q-52 0 -90 38t-38 90v307l-266 -153q-46 -26 -97.5 -12.5t-77.5 59.5l-64 110q-26 46 -12.5 97.5t59.5 77.5l266 154l-266 154 q-46 26 -59.5 77.5t12.5 97.5l64 110q26 46 77.5 59.5t97.5 -12.5l266 -153v307q0 52 38 90t90 38h128q52 0 90 -38t38 -90v-307l266 153q46 26 97.5 12.5t77.5 -59.5l64 -110q26 -46 12.5 -97.5t-59.5 -77.5l-266 -154z" />
+<glyph unicode="" d="M768 1408q209 0 385.5 -103t279.5 -279.5t103 -385.5t-103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103zM896 161v190q0 14 -9 23.5t-22 9.5h-192q-13 0 -23 -10t-10 -23v-190q0 -13 10 -23t23 -10h192 q13 0 22 9.5t9 23.5zM894 505l18 621q0 12 -10 18q-10 8 -24 8h-220q-14 0 -24 -8q-10 -6 -10 -18l17 -621q0 -10 10 -17.5t24 -7.5h185q14 0 23.5 7.5t10.5 17.5z" />
+<glyph unicode="" d="M928 180v56v468v192h-320v-192v-468v-56q0 -25 18 -38.5t46 -13.5h192q28 0 46 13.5t18 38.5zM472 1024h195l-126 161q-26 31 -69 31q-40 0 -68 -28t-28 -68t28 -68t68 -28zM1160 1120q0 40 -28 68t-68 28q-43 0 -69 -31l-125 -161h194q40 0 68 28t28 68zM1536 864v-320 q0 -14 -9 -23t-23 -9h-96v-416q0 -40 -28 -68t-68 -28h-1088q-40 0 -68 28t-28 68v416h-96q-14 0 -23 9t-9 23v320q0 14 9 23t23 9h440q-93 0 -158.5 65.5t-65.5 158.5t65.5 158.5t158.5 65.5q107 0 168 -77l128 -165l128 165q61 77 168 77q93 0 158.5 -65.5t65.5 -158.5 t-65.5 -158.5t-158.5 -65.5h440q14 0 23 -9t9 -23z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1280 832q0 26 -19 45t-45 19q-172 0 -318 -49.5t-259.5 -134t-235.5 -219.5q-19 -21 -19 -45q0 -26 19 -45t45 -19q24 0 45 19q27 24 74 71t67 66q137 124 268.5 176t313.5 52q26 0 45 19t19 45zM1792 1030q0 -95 -20 -193q-46 -224 -184.5 -383t-357.5 -268 q-214 -108 -438 -108q-148 0 -286 47q-15 5 -88 42t-96 37q-16 0 -39.5 -32t-45 -70t-52.5 -70t-60 -32q-30 0 -51 11t-31 24t-27 42q-2 4 -6 11t-5.5 10t-3 9.5t-1.5 13.5q0 35 31 73.5t68 65.5t68 56t31 48q0 4 -14 38t-16 44q-9 51 -9 104q0 115 43.5 220t119 184.5 t170.5 139t204 95.5q55 18 145 25.5t179.5 9t178.5 6t163.5 24t113.5 56.5l29.5 29.5t29.5 28t27 20t36.5 16t43.5 4.5q39 0 70.5 -46t47.5 -112t24 -124t8 -96z" />
+<glyph unicode="" horiz-adv-x="1408" d="M1408 -160v-64q0 -13 -9.5 -22.5t-22.5 -9.5h-1344q-13 0 -22.5 9.5t-9.5 22.5v64q0 13 9.5 22.5t22.5 9.5h1344q13 0 22.5 -9.5t9.5 -22.5zM1152 896q0 -78 -24.5 -144t-64 -112.5t-87.5 -88t-96 -77.5t-87.5 -72t-64 -81.5t-24.5 -96.5q0 -96 67 -224l-4 1l1 -1 q-90 41 -160 83t-138.5 100t-113.5 122.5t-72.5 150.5t-27.5 184q0 78 24.5 144t64 112.5t87.5 88t96 77.5t87.5 72t64 81.5t24.5 96.5q0 94 -66 224l3 -1l-1 1q90 -41 160 -83t138.5 -100t113.5 -122.5t72.5 -150.5t27.5 -184z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1664 576q-152 236 -381 353q61 -104 61 -225q0 -185 -131.5 -316.5t-316.5 -131.5t-316.5 131.5t-131.5 316.5q0 121 61 225q-229 -117 -381 -353q133 -205 333.5 -326.5t434.5 -121.5t434.5 121.5t333.5 326.5zM944 960q0 20 -14 34t-34 14q-125 0 -214.5 -89.5 t-89.5 -214.5q0 -20 14 -34t34 -14t34 14t14 34q0 86 61 147t147 61q20 0 34 14t14 34zM1792 576q0 -34 -20 -69q-140 -230 -376.5 -368.5t-499.5 -138.5t-499.5 139t-376.5 368q-20 35 -20 69t20 69q140 229 376.5 368t499.5 139t499.5 -139t376.5 -368q20 -35 20 -69z" />
+<glyph unicode="" horiz-adv-x="1792" d="M555 201l78 141q-87 63 -136 159t-49 203q0 121 61 225q-229 -117 -381 -353q167 -258 427 -375zM944 960q0 20 -14 34t-34 14q-125 0 -214.5 -89.5t-89.5 -214.5q0 -20 14 -34t34 -14t34 14t14 34q0 86 61 147t147 61q20 0 34 14t14 34zM1307 1151q0 -7 -1 -9 q-105 -188 -315 -566t-316 -567l-49 -89q-10 -16 -28 -16q-12 0 -134 70q-16 10 -16 28q0 12 44 87q-143 65 -263.5 173t-208.5 245q-20 31 -20 69t20 69q153 235 380 371t496 136q89 0 180 -17l54 97q10 16 28 16q5 0 18 -6t31 -15.5t33 -18.5t31.5 -18.5t19.5 -11.5 q16 -10 16 -27zM1344 704q0 -139 -79 -253.5t-209 -164.5l280 502q8 -45 8 -84zM1792 576q0 -35 -20 -69q-39 -64 -109 -145q-150 -172 -347.5 -267t-419.5 -95l74 132q212 18 392.5 137t301.5 307q-115 179 -282 294l63 112q95 -64 182.5 -153t144.5 -184q20 -34 20 -69z " />
+<glyph unicode="" horiz-adv-x="1792" d="M1024 161v190q0 14 -9.5 23.5t-22.5 9.5h-192q-13 0 -22.5 -9.5t-9.5 -23.5v-190q0 -14 9.5 -23.5t22.5 -9.5h192q13 0 22.5 9.5t9.5 23.5zM1022 535l18 459q0 12 -10 19q-13 11 -24 11h-220q-11 0 -24 -11q-10 -7 -10 -21l17 -457q0 -10 10 -16.5t24 -6.5h185 q14 0 23.5 6.5t10.5 16.5zM1008 1469l768 -1408q35 -63 -2 -126q-17 -29 -46.5 -46t-63.5 -17h-1536q-34 0 -63.5 17t-46.5 46q-37 63 -2 126l768 1408q17 31 47 49t65 18t65 -18t47 -49z" />
+<glyph unicode="" horiz-adv-x="1408" d="M1376 1376q44 -52 12 -148t-108 -172l-161 -161l160 -696q5 -19 -12 -33l-128 -96q-7 -6 -19 -6q-4 0 -7 1q-15 3 -21 16l-279 508l-259 -259l53 -194q5 -17 -8 -31l-96 -96q-9 -9 -23 -9h-2q-15 2 -24 13l-189 252l-252 189q-11 7 -13 23q-1 13 9 25l96 97q9 9 23 9 q6 0 8 -1l194 -53l259 259l-508 279q-14 8 -17 24q-2 16 9 27l128 128q14 13 30 8l665 -159l160 160q76 76 172 108t148 -12z" />
+<glyph unicode="" horiz-adv-x="1664" d="M128 -128h288v288h-288v-288zM480 -128h320v288h-320v-288zM128 224h288v320h-288v-320zM480 224h320v320h-320v-320zM128 608h288v288h-288v-288zM864 -128h320v288h-320v-288zM480 608h320v288h-320v-288zM1248 -128h288v288h-288v-288zM864 224h320v320h-320v-320z M512 1088v288q0 13 -9.5 22.5t-22.5 9.5h-64q-13 0 -22.5 -9.5t-9.5 -22.5v-288q0 -13 9.5 -22.5t22.5 -9.5h64q13 0 22.5 9.5t9.5 22.5zM1248 224h288v320h-288v-320zM864 608h320v288h-320v-288zM1248 608h288v288h-288v-288zM1280 1088v288q0 13 -9.5 22.5t-22.5 9.5h-64 q-13 0 -22.5 -9.5t-9.5 -22.5v-288q0 -13 9.5 -22.5t22.5 -9.5h64q13 0 22.5 9.5t9.5 22.5zM1664 1152v-1280q0 -52 -38 -90t-90 -38h-1408q-52 0 -90 38t-38 90v1280q0 52 38 90t90 38h128v96q0 66 47 113t113 47h64q66 0 113 -47t47 -113v-96h384v96q0 66 47 113t113 47 h64q66 0 113 -47t47 -113v-96h128q52 0 90 -38t38 -90z" />
+<glyph unicode="" horiz-adv-x="1792" d="M666 1055q-60 -92 -137 -273q-22 45 -37 72.5t-40.5 63.5t-51 56.5t-63 35t-81.5 14.5h-224q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h224q250 0 410 -225zM1792 256q0 -14 -9 -23l-320 -320q-9 -9 -23 -9q-13 0 -22.5 9.5t-9.5 22.5v192q-32 0 -85 -0.5t-81 -1t-73 1 t-71 5t-64 10.5t-63 18.5t-58 28.5t-59 40t-55 53.5t-56 69.5q59 93 136 273q22 -45 37 -72.5t40.5 -63.5t51 -56.5t63 -35t81.5 -14.5h256v192q0 14 9 23t23 9q12 0 24 -10l319 -319q9 -9 9 -23zM1792 1152q0 -14 -9 -23l-320 -320q-9 -9 -23 -9q-13 0 -22.5 9.5t-9.5 22.5 v192h-256q-48 0 -87 -15t-69 -45t-51 -61.5t-45 -77.5q-32 -62 -78 -171q-29 -66 -49.5 -111t-54 -105t-64 -100t-74 -83t-90 -68.5t-106.5 -42t-128 -16.5h-224q-14 0 -23 9t-9 23v192q0 14 9 23t23 9h224q48 0 87 15t69 45t51 61.5t45 77.5q32 62 78 171q29 66 49.5 111 t54 105t64 100t74 83t90 68.5t106.5 42t128 16.5h256v192q0 14 9 23t23 9q12 0 24 -10l319 -319q9 -9 9 -23z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1792 640q0 -174 -120 -321.5t-326 -233t-450 -85.5q-70 0 -145 8q-198 -175 -460 -242q-49 -14 -114 -22q-17 -2 -30.5 9t-17.5 29v1q-3 4 -0.5 12t2 10t4.5 9.5l6 9t7 8.5t8 9q7 8 31 34.5t34.5 38t31 39.5t32.5 51t27 59t26 76q-157 89 -247.5 220t-90.5 281 q0 130 71 248.5t191 204.5t286 136.5t348 50.5q244 0 450 -85.5t326 -233t120 -321.5z" />
+<glyph unicode="" d="M1536 704v-128q0 -201 -98.5 -362t-274 -251.5t-395.5 -90.5t-395.5 90.5t-274 251.5t-98.5 362v128q0 26 19 45t45 19h384q26 0 45 -19t19 -45v-128q0 -52 23.5 -90t53.5 -57t71 -30t64 -13t44 -2t44 2t64 13t71 30t53.5 57t23.5 90v128q0 26 19 45t45 19h384 q26 0 45 -19t19 -45zM512 1344v-384q0 -26 -19 -45t-45 -19h-384q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h384q26 0 45 -19t19 -45zM1536 1344v-384q0 -26 -19 -45t-45 -19h-384q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h384q26 0 45 -19t19 -45z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1683 205l-166 -165q-19 -19 -45 -19t-45 19l-531 531l-531 -531q-19 -19 -45 -19t-45 19l-166 165q-19 19 -19 45.5t19 45.5l742 741q19 19 45 19t45 -19l742 -741q19 -19 19 -45.5t-19 -45.5z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1683 728l-742 -741q-19 -19 -45 -19t-45 19l-742 741q-19 19 -19 45.5t19 45.5l166 165q19 19 45 19t45 -19l531 -531l531 531q19 19 45 19t45 -19l166 -165q19 -19 19 -45.5t-19 -45.5z" />
+<glyph unicode="" horiz-adv-x="1920" d="M1280 32q0 -13 -9.5 -22.5t-22.5 -9.5h-960q-8 0 -13.5 2t-9 7t-5.5 8t-3 11.5t-1 11.5v13v11v160v416h-192q-26 0 -45 19t-19 45q0 24 15 41l320 384q19 22 49 22t49 -22l320 -384q15 -17 15 -41q0 -26 -19 -45t-45 -19h-192v-384h576q16 0 25 -11l160 -192q7 -11 7 -21 zM1920 448q0 -24 -15 -41l-320 -384q-20 -23 -49 -23t-49 23l-320 384q-15 17 -15 41q0 26 19 45t45 19h192v384h-576q-16 0 -25 12l-160 192q-7 9 -7 20q0 13 9.5 22.5t22.5 9.5h960q8 0 13.5 -2t9 -7t5.5 -8t3 -11.5t1 -11.5v-13v-11v-160v-416h192q26 0 45 -19t19 -45z " />
+<glyph unicode="" horiz-adv-x="1664" d="M640 0q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1536 0q0 -53 -37.5 -90.5t-90.5 -37.5t-90.5 37.5t-37.5 90.5t37.5 90.5t90.5 37.5t90.5 -37.5t37.5 -90.5zM1664 1088v-512q0 -24 -16 -42.5t-41 -21.5 l-1044 -122q1 -7 4.5 -21.5t6 -26.5t2.5 -22q0 -16 -24 -64h920q26 0 45 -19t19 -45t-19 -45t-45 -19h-1024q-26 0 -45 19t-19 45q0 14 11 39.5t29.5 59.5t20.5 38l-177 823h-204q-26 0 -45 19t-19 45t19 45t45 19h256q16 0 28.5 -6.5t20 -15.5t13 -24.5t7.5 -26.5 t5.5 -29.5t4.5 -25.5h1201q26 0 45 -19t19 -45z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1664 928v-704q0 -92 -66 -158t-158 -66h-1216q-92 0 -158 66t-66 158v960q0 92 66 158t158 66h320q92 0 158 -66t66 -158v-32h672q92 0 158 -66t66 -158z" />
+<glyph unicode="" horiz-adv-x="1920" d="M1879 584q0 -31 -31 -66l-336 -396q-43 -51 -120.5 -86.5t-143.5 -35.5h-1088q-34 0 -60.5 13t-26.5 43q0 31 31 66l336 396q43 51 120.5 86.5t143.5 35.5h1088q34 0 60.5 -13t26.5 -43zM1536 928v-160h-832q-94 0 -197 -47.5t-164 -119.5l-337 -396l-5 -6q0 4 -0.5 12.5 t-0.5 12.5v960q0 92 66 158t158 66h320q92 0 158 -66t66 -158v-32h544q92 0 158 -66t66 -158z" />
+<glyph unicode="" horiz-adv-x="768" d="M704 1216q0 -26 -19 -45t-45 -19h-128v-1024h128q26 0 45 -19t19 -45t-19 -45l-256 -256q-19 -19 -45 -19t-45 19l-256 256q-19 19 -19 45t19 45t45 19h128v1024h-128q-26 0 -45 19t-19 45t19 45l256 256q19 19 45 19t45 -19l256 -256q19 -19 19 -45z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1792 640q0 -26 -19 -45l-256 -256q-19 -19 -45 -19t-45 19t-19 45v128h-1024v-128q0 -26 -19 -45t-45 -19t-45 19l-256 256q-19 19 -19 45t19 45l256 256q19 19 45 19t45 -19t19 -45v-128h1024v128q0 26 19 45t45 19t45 -19l256 -256q19 -19 19 -45z" />
+<glyph unicode="" horiz-adv-x="2048" d="M640 640v-512h-256v512h256zM1024 1152v-1024h-256v1024h256zM2048 0v-128h-2048v1536h128v-1408h1920zM1408 896v-768h-256v768h256zM1792 1280v-1152h-256v1152h256z" />
+<glyph unicode="" d="M1280 926q-56 -25 -121 -34q68 40 93 117q-65 -38 -134 -51q-61 66 -153 66q-87 0 -148.5 -61.5t-61.5 -148.5q0 -29 5 -48q-129 7 -242 65t-192 155q-29 -50 -29 -106q0 -114 91 -175q-47 1 -100 26v-2q0 -75 50 -133.5t123 -72.5q-29 -8 -51 -8q-13 0 -39 4 q21 -63 74.5 -104t121.5 -42q-116 -90 -261 -90q-26 0 -50 3q148 -94 322 -94q112 0 210 35.5t168 95t120.5 137t75 162t24.5 168.5q0 18 -1 27q63 45 105 109zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5 t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="" d="M1536 160q0 -119 -84.5 -203.5t-203.5 -84.5h-192v608h203l30 224h-233v143q0 54 28 83t96 29l132 1v207q-96 9 -180 9q-136 0 -218 -80.5t-82 -225.5v-166h-224v-224h224v-608h-544q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960 q119 0 203.5 -84.5t84.5 -203.5v-960z" />
+<glyph unicode="" horiz-adv-x="1792" d="M928 704q0 14 -9 23t-23 9q-66 0 -113 -47t-47 -113q0 -14 9 -23t23 -9t23 9t9 23q0 40 28 68t68 28q14 0 23 9t9 23zM1152 574q0 -106 -75 -181t-181 -75t-181 75t-75 181t75 181t181 75t181 -75t75 -181zM128 0h1536v128h-1536v-128zM1280 574q0 159 -112.5 271.5 t-271.5 112.5t-271.5 -112.5t-112.5 -271.5t112.5 -271.5t271.5 -112.5t271.5 112.5t112.5 271.5zM256 1216h384v128h-384v-128zM128 1024h1536v118v138h-828l-64 -128h-644v-128zM1792 1280v-1280q0 -53 -37.5 -90.5t-90.5 -37.5h-1536q-53 0 -90.5 37.5t-37.5 90.5v1280 q0 53 37.5 90.5t90.5 37.5h1536q53 0 90.5 -37.5t37.5 -90.5z" />
+<glyph unicode="" horiz-adv-x="1792" d="M832 1024q0 80 -56 136t-136 56t-136 -56t-56 -136q0 -42 19 -83q-41 19 -83 19q-80 0 -136 -56t-56 -136t56 -136t136 -56t136 56t56 136q0 42 -19 83q41 -19 83 -19q80 0 136 56t56 136zM1683 320q0 -17 -49 -66t-66 -49q-9 0 -28.5 16t-36.5 33t-38.5 40t-24.5 26 l-96 -96l220 -220q28 -28 28 -68q0 -42 -39 -81t-81 -39q-40 0 -68 28l-671 671q-176 -131 -365 -131q-163 0 -265.5 102.5t-102.5 265.5q0 160 95 313t248 248t313 95q163 0 265.5 -102.5t102.5 -265.5q0 -189 -131 -365l355 -355l96 96q-3 3 -26 24.5t-40 38.5t-33 36.5 t-16 28.5q0 17 49 66t66 49q13 0 23 -10q6 -6 46 -44.5t82 -79.5t86.5 -86t73 -78t28.5 -41z" />
+<glyph unicode="" horiz-adv-x="1920" d="M896 640q0 106 -75 181t-181 75t-181 -75t-75 -181t75 -181t181 -75t181 75t75 181zM1664 128q0 52 -38 90t-90 38t-90 -38t-38 -90q0 -53 37.5 -90.5t90.5 -37.5t90.5 37.5t37.5 90.5zM1664 1152q0 52 -38 90t-90 38t-90 -38t-38 -90q0 -53 37.5 -90.5t90.5 -37.5 t90.5 37.5t37.5 90.5zM1280 731v-185q0 -10 -7 -19.5t-16 -10.5l-155 -24q-11 -35 -32 -76q34 -48 90 -115q7 -10 7 -20q0 -12 -7 -19q-23 -30 -82.5 -89.5t-78.5 -59.5q-11 0 -21 7l-115 90q-37 -19 -77 -31q-11 -108 -23 -155q-7 -24 -30 -24h-186q-11 0 -20 7.5t-10 17.5 l-23 153q-34 10 -75 31l-118 -89q-7 -7 -20 -7q-11 0 -21 8q-144 133 -144 160q0 9 7 19q10 14 41 53t47 61q-23 44 -35 82l-152 24q-10 1 -17 9.5t-7 19.5v185q0 10 7 19.5t16 10.5l155 24q11 35 32 76q-34 48 -90 115q-7 11 -7 20q0 12 7 20q22 30 82 89t79 59q11 0 21 -7 l115 -90q34 18 77 32q11 108 23 154q7 24 30 24h186q11 0 20 -7.5t10 -17.5l23 -153q34 -10 75 -31l118 89q8 7 20 7q11 0 21 -8q144 -133 144 -160q0 -9 -7 -19q-12 -16 -42 -54t-45 -60q23 -48 34 -82l152
-23q10 -2 17 -10.5t7 -19.5zM1920 198v-140q0 -16 -149 -31 q-12 -27 -30 -52q51 -113 51 -138q0 -4 -4 -7q-122 -71 -124 -71q-8 0 -46 47t-52 68q-20 -2 -30 -2t-30 2q-14 -21 -52 -68t-46 -47q-2 0 -124 71q-4 3 -4 7q0 25 51 138q-18 25 -30 52q-149 15 -149 31v140q0 16 149 31q13 29 30 52q-51 113 -51 138q0 4 4 7q4 2 35 20 t59 34t30 16q8 0 46 -46.5t52 -67.5q20 2 30 2t30 -2q51 71 92 112l6 2q4 0 124 -70q4 -3 4 -7q0 -25 -51 -138q17 -23 30 -52q149 -15 149 -31zM1920 1222v-140q0 -16 -149 -31q-12 -27 -30 -52q51 -113 51 -138q0 -4 -4 -7q-122 -71 -124 -71q-8 0 -46 47t-52 68 q-20 -2 -30 -2t-30 2q-14 -21 -52 -68t-46 -47q-2 0 -124 71q-4 3 -4 7q0 25 51 138q-18 25 -30 52q-149 15 -149 31v140q0 16 149 31q13 29 30 52q-51 113 -51 138q0 4 4 7q4 2 35 20t59 34t30 16q8 0 46 -46.5t52 -67.5q20 2 30 2t30 -2q51 71 92 112l6 2q4 0 124 -70 q4 -3 4 -7q0 -25 -51 -138q17 -23 30 -52q149 -15 149 -31z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1408 768q0 -139 -94 -257t-256.5 -186.5t-353.5 -68.5q-86 0 -176 16q-124 -88 -278 -128q-36 -9 -86 -16h-3q-11 0 -20.5 8t-11.5 21q-1 3 -1 6.5t0.5 6.5t2 6l2.5 5t3.5 5.5t4 5t4.5 5t4 4.5q5 6 23 25t26 29.5t22.5 29t25 38.5t20.5 44q-124 72 -195 177t-71 224 q0 139 94 257t256.5 186.5t353.5 68.5t353.5 -68.5t256.5 -186.5t94 -257zM1792 512q0 -120 -71 -224.5t-195 -176.5q10 -24 20.5 -44t25 -38.5t22.5 -29t26 -29.5t23 -25q1 -1 4 -4.5t4.5 -5t4 -5t3.5 -5.5l2.5 -5t2 -6t0.5 -6.5t-1 -6.5q-3 -14 -13 -22t-22 -7 q-50 7 -86 16q-154 40 -278 128q-90 -16 -176 -16q-271 0 -472 132q58 -4 88 -4q161 0 309 45t264 129q125 92 192 212t67 254q0 77 -23 152q129 -71 204 -178t75 -230z" />
+<glyph unicode="" d="M256 192q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1408 768q0 51 -39 89.5t-89 38.5h-352q0 58 48 159.5t48 160.5q0 98 -32 145t-128 47q-26 -26 -38 -85t-30.5 -125.5t-59.5 -109.5q-22 -23 -77 -91q-4 -5 -23 -30t-31.5 -41t-34.5 -42.5 t-40 -44t-38.5 -35.5t-40 -27t-35.5 -9h-32v-640h32q13 0 31.5 -3t33 -6.5t38 -11t35 -11.5t35.5 -12.5t29 -10.5q211 -73 342 -73h121q192 0 192 167q0 26 -5 56q30 16 47.5 52.5t17.5 73.5t-18 69q53 50 53 119q0 25 -10 55.5t-25 47.5q32 1 53.5 47t21.5 81zM1536 769 q0 -89 -49 -163q9 -33 9 -69q0 -77 -38 -144q3 -21 3 -43q0 -101 -60 -178q1 -139 -85 -219.5t-227 -80.5h-36h-93q-96 0 -189.5 22.5t-216.5 65.5q-116 40 -138 40h-288q-53 0 -90.5 37.5t-37.5 90.5v640q0 53 37.5 90.5t90.5 37.5h274q36 24 137 155q58 75 107 128 q24 25 35.5 85.5t30.5 126.5t62 108q39 37 90 37q84 0 151 -32.5t102 -101.5t35 -186q0 -93 -48 -192h176q104 0 180 -76t76 -179z" />
+<glyph unicode="" d="M256 1088q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1408 512q0 35 -21.5 81t-53.5 47q15 17 25 47.5t10 55.5q0 69 -53 119q18 32 18 69t-17.5 73.5t-47.5 52.5q5 30 5 56q0 85 -49 126t-136 41h-128q-131 0 -342 -73q-5 -2 -29 -10.5 t-35.5 -12.5t-35 -11.5t-38 -11t-33 -6.5t-31.5 -3h-32v-640h32q16 0 35.5 -9t40 -27t38.5 -35.5t40 -44t34.5 -42.5t31.5 -41t23 -30q55 -68 77 -91q41 -43 59.5 -109.5t30.5 -125.5t38 -85q96 0 128 47t32 145q0 59 -48 160.5t-48 159.5h352q50 0 89 38.5t39 89.5z M1536 511q0 -103 -76 -179t-180 -76h-176q48 -99 48 -192q0 -118 -35 -186q-35 -69 -102 -101.5t-151 -32.5q-51 0 -90 37q-34 33 -54 82t-25.5 90.5t-17.5 84.5t-31 64q-48 50 -107 127q-101 131 -137 155h-274q-53 0 -90.5 37.5t-37.5 90.5v640q0 53 37.5 90.5t90.5 37.5 h288q22 0 138 40q128 44 223 66t200 22h112q140 0 226.5 -79t85.5 -216v-5q60 -77 60 -178q0 -22 -3 -43q38 -67 38 -144q0 -36 -9 -69q49 -74 49 -163z" />
+<glyph unicode="" horiz-adv-x="896" d="M832 1504v-1339l-449 -236q-22 -12 -40 -12q-21 0 -31.5 14.5t-10.5 35.5q0 6 2 20l86 500l-364 354q-25 27 -25 48q0 37 56 46l502 73l225 455q19 41 49 41z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1664 940q0 81 -21.5 143t-55 98.5t-81.5 59.5t-94 31t-98 8t-112 -25.5t-110.5 -64t-86.5 -72t-60 -61.5q-18 -22 -49 -22t-49 22q-24 28 -60 61.5t-86.5 72t-110.5 64t-112 25.5t-98 -8t-94 -31t-81.5 -59.5t-55 -98.5t-21.5 -143q0 -168 187 -355l581 -560l580 559 q188 188 188 356zM1792 940q0 -221 -229 -450l-623 -600q-18 -18 -44 -18t-44 18l-624 602q-10 8 -27.5 26t-55.5 65.5t-68 97.5t-53.5 121t-23.5 138q0 220 127 344t351 124q62 0 126.5 -21.5t120 -58t95.5 -68.5t76 -68q36 36 76 68t95.5 68.5t120 58t126.5 21.5 q224 0 351 -124t127 -344z" />
+<glyph unicode="" horiz-adv-x="1664" d="M640 96q0 -4 1 -20t0.5 -26.5t-3 -23.5t-10 -19.5t-20.5 -6.5h-320q-119 0 -203.5 84.5t-84.5 203.5v704q0 119 84.5 203.5t203.5 84.5h320q13 0 22.5 -9.5t9.5 -22.5q0 -4 1 -20t0.5 -26.5t-3 -23.5t-10 -19.5t-20.5 -6.5h-320q-66 0 -113 -47t-47 -113v-704 q0 -66 47 -113t113 -47h288h11h13t11.5 -1t11.5 -3t8 -5.5t7 -9t2 -13.5zM1568 640q0 -26 -19 -45l-544 -544q-19 -19 -45 -19t-45 19t-19 45v288h-448q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h448v288q0 26 19 45t45 19t45 -19l544 -544q19 -19 19 -45z" />
+<glyph unicode="" d="M237 122h231v694h-231v-694zM483 1030q-1 52 -36 86t-93 34t-94.5 -34t-36.5 -86q0 -51 35.5 -85.5t92.5 -34.5h1q59 0 95 34.5t36 85.5zM1068 122h231v398q0 154 -73 233t-193 79q-136 0 -209 -117h2v101h-231q3 -66 0 -694h231v388q0 38 7 56q15 35 45 59.5t74 24.5 q116 0 116 -157v-371zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="" horiz-adv-x="1152" d="M480 672v448q0 14 -9 23t-23 9t-23 -9t-9 -23v-448q0 -14 9 -23t23 -9t23 9t9 23zM1152 320q0 -26 -19 -45t-45 -19h-429l-51 -483q-2 -12 -10.5 -20.5t-20.5 -8.5h-1q-27 0 -32 27l-76 485h-404q-26 0 -45 19t-19 45q0 123 78.5 221.5t177.5 98.5v512q-52 0 -90 38 t-38 90t38 90t90 38h640q52 0 90 -38t38 -90t-38 -90t-90 -38v-512q99 0 177.5 -98.5t78.5 -221.5z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1408 608v-320q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h704q14 0 23 -9t9 -23v-64q0 -14 -9 -23t-23 -9h-704q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832q66 0 113 47t47 113v320 q0 14 9 23t23 9h64q14 0 23 -9t9 -23zM1792 1472v-512q0 -26 -19 -45t-45 -19t-45 19l-176 176l-652 -652q-10 -10 -23 -10t-23 10l-114 114q-10 10 -10 23t10 23l652 652l-176 176q-19 19 -19 45t19 45t45 19h512q26 0 45 -19t19 -45z" />
+<glyph unicode="" d="M1184 640q0 -26 -19 -45l-544 -544q-19 -19 -45 -19t-45 19t-19 45v288h-448q-26 0 -45 19t-19 45v384q0 26 19 45t45 19h448v288q0 26 19 45t45 19t45 -19l544 -544q19 -19 19 -45zM1536 992v-704q0 -119 -84.5 -203.5t-203.5 -84.5h-320q-13 0 -22.5 9.5t-9.5 22.5 q0 4 -1 20t-0.5 26.5t3 23.5t10 19.5t20.5 6.5h320q66 0 113 47t47 113v704q0 66 -47 113t-113 47h-288h-11h-13t-11.5 1t-11.5 3t-8 5.5t-7 9t-2 13.5q0 4 -1 20t-0.5 26.5t3 23.5t10 19.5t20.5 6.5h320q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="" horiz-adv-x="1664" d="M458 653q-74 162 -74 371h-256v-96q0 -78 94.5 -162t235.5 -113zM1536 928v96h-256q0 -209 -74 -371q141 29 235.5 113t94.5 162zM1664 1056v-128q0 -71 -41.5 -143t-112 -130t-173 -97.5t-215.5 -44.5q-42 -54 -95 -95q-38 -34 -52.5 -72.5t-14.5 -89.5q0 -54 30.5 -91 t97.5 -37q75 0 133.5 -45.5t58.5 -114.5v-64q0 -14 -9 -23t-23 -9h-832q-14 0 -23 9t-9 23v64q0 69 58.5 114.5t133.5 45.5q67 0 97.5 37t30.5 91q0 51 -14.5 89.5t-52.5 72.5q-53 41 -95 95q-113 5 -215.5 44.5t-173 97.5t-112 130t-41.5 143v128q0 40 28 68t68 28h288v96 q0 66 47 113t113 47h576q66 0 113 -47t47 -113v-96h288q40 0 68 -28t28 -68z" />
+<glyph unicode="" d="M394 184q-8 -9 -20 3q-13 11 -4 19q8 9 20 -3q12 -11 4 -19zM352 245q9 -12 0 -19q-8 -6 -17 7t0 18q9 7 17 -6zM291 305q-5 -7 -13 -2q-10 5 -7 12q3 5 13 2q10 -5 7 -12zM322 271q-6 -7 -16 3q-9 11 -2 16q6 6 16 -3q9 -11 2 -16zM451 159q-4 -12 -19 -6q-17 4 -13 15 t19 7q16 -5 13 -16zM514 154q0 -11 -16 -11q-17 -2 -17 11q0 11 16 11q17 2 17 -11zM572 164q2 -10 -14 -14t-18 8t14 15q16 2 18 -9zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-224q-16 0 -24.5 1t-19.5 5t-16 14.5t-5 27.5v239q0 97 -52 142q57 6 102.5 18t94 39 t81 66.5t53 105t20.5 150.5q0 121 -79 206q37 91 -8 204q-28 9 -81 -11t-92 -44l-38 -24q-93 26 -192 26t-192 -26q-16 11 -42.5 27t-83.5 38.5t-86 13.5q-44 -113 -7 -204q-79 -85 -79 -206q0 -85 20.5 -150t52.5 -105t80.5 -67t94 -39t102.5 -18q-40 -36 -49 -103 q-21 -10 -45 -15t-57 -5t-65.5 21.5t-55.5 62.5q-19 32 -48.5 52t-49.5 24l-20 3q-21 0 -29 -4.5t-5 -11.5t9 -14t13 -12l7 -5q22 -10 43.5 -38t31.5 -51l10 -23q13 -38 44 -61.5t67 -30t69.5 -7t55.5 3.5l23 4q0 -38 0.5 -103t0.5
-68q0 -22 -11 -33.5t-22 -13t-33 -1.5 h-224q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1280 64q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1536 64q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1664 288v-320q0 -40 -28 -68t-68 -28h-1472q-40 0 -68 28t-28 68v320q0 40 28 68t68 28h427q21 -56 70.5 -92 t110.5 -36h256q61 0 110.5 36t70.5 92h427q40 0 68 -28t28 -68zM1339 936q-17 -40 -59 -40h-256v-448q0 -26 -19 -45t-45 -19h-256q-26 0 -45 19t-19 45v448h-256q-42 0 -59 40q-17 39 14 69l448 448q18 19 45 19t45 -19l448 -448q31 -30 14 -69z" />
+<glyph unicode="" d="M1407 710q0 44 -7 113.5t-18 96.5q-12 30 -17 44t-9 36.5t-4 48.5q0 23 5 68.5t5 67.5q0 37 -10 55q-4 1 -13 1q-19 0 -58 -4.5t-59 -4.5q-60 0 -176 24t-175 24q-43 0 -94.5 -11.5t-85 -23.5t-89.5 -34q-137 -54 -202 -103q-96 -73 -159.5 -189.5t-88 -236t-24.5 -248.5 q0 -40 12.5 -120t12.5 -121q0 -23 -11 -66.5t-11 -65.5t12 -36.5t34 -14.5q24 0 72.5 11t73.5 11q57 0 169.5 -15.5t169.5 -15.5q181 0 284 36q129 45 235.5 152.5t166 245.5t59.5 275zM1535 712q0 -165 -70 -327.5t-196 -288t-281 -180.5q-124 -44 -326 -44 q-57 0 -170 14.5t-169 14.5q-24 0 -72.5 -14.5t-73.5 -14.5q-73 0 -123.5 55.5t-50.5 128.5q0 24 11 68t11 67q0 40 -12.5 120.5t-12.5 121.5q0 111 18 217.5t54.5 209.5t100.5 194t150 156q78 59 232 120q194 78 316 78q60 0 175.5 -24t173.5 -24q19 0 57 5t58 5 q81 0 118 -50.5t37 -134.5q0 -23 -5 -68t-5 -68q0 -10 1 -18.5t3 -17t4 -13.5t6.5 -16t6.5 -17q16 -40 25 -118.5t9 -136.5z" />
+<glyph unicode="" horiz-adv-x="1408" d="M1408 296q0 -27 -10 -70.5t-21 -68.5q-21 -50 -122 -106q-94 -51 -186 -51q-27 0 -52.5 3.5t-57.5 12.5t-47.5 14.5t-55.5 20.5t-49 18q-98 35 -175 83q-128 79 -264.5 215.5t-215.5 264.5q-48 77 -83 175q-3 9 -18 49t-20.5 55.5t-14.5 47.5t-12.5 57.5t-3.5 52.5 q0 92 51 186q56 101 106 122q25 11 68.5 21t70.5 10q14 0 21 -3q18 -6 53 -76q11 -19 30 -54t35 -63.5t31 -53.5q3 -4 17.5 -25t21.5 -35.5t7 -28.5q0 -20 -28.5 -50t-62 -55t-62 -53t-28.5 -46q0 -9 5 -22.5t8.5 -20.5t14 -24t11.5 -19q76 -137 174 -235t235 -174 q2 -1 19 -11.5t24 -14t20.5 -8.5t22.5 -5q18 0 46 28.5t53 62t55 62t50 28.5q14 0 28.5 -7t35.5 -21.5t25 -17.5q25 -15 53.5 -31t63.5 -35t54 -30q70 -35 76 -53q3 -7 3 -21z" />
+<glyph unicode="" horiz-adv-x="1408" d="M1120 1280h-832q-66 0 -113 -47t-47 -113v-832q0 -66 47 -113t113 -47h832q66 0 113 47t47 113v832q0 66 -47 113t-113 47zM1408 1120v-832q0 -119 -84.5 -203.5t-203.5 -84.5h-832q-119 0 -203.5 84.5t-84.5 203.5v832q0 119 84.5 203.5t203.5 84.5h832 q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="" horiz-adv-x="1280" d="M1152 1280h-1024v-1242l423 406l89 85l89 -85l423 -406v1242zM1164 1408q23 0 44 -9q33 -13 52.5 -41t19.5 -62v-1289q0 -34 -19.5 -62t-52.5 -41q-19 -8 -44 -8q-48 0 -83 32l-441 424l-441 -424q-36 -33 -83 -33q-23 0 -44 9q-33 13 -52.5 41t-19.5 62v1289 q0 34 19.5 62t52.5 41q21 9 44 9h1048z" />
+<glyph unicode="" d="M1280 343q0 11 -2 16q-3 8 -38.5 29.5t-88.5 49.5l-53 29q-5 3 -19 13t-25 15t-21 5q-18 0 -47 -32.5t-57 -65.5t-44 -33q-7 0 -16.5 3.5t-15.5 6.5t-17 9.5t-14 8.5q-99 55 -170.5 126.5t-126.5 170.5q-2 3 -8.5 14t-9.5 17t-6.5 15.5t-3.5 16.5q0 13 20.5 33.5t45 38.5 t45 39.5t20.5 36.5q0 10 -5 21t-15 25t-13 19q-3 6 -15 28.5t-25 45.5t-26.5 47.5t-25 40.5t-16.5 18t-16 2q-48 0 -101 -22q-46 -21 -80 -94.5t-34 -130.5q0 -16 2.5 -34t5 -30.5t9 -33t10 -29.5t12.5 -33t11 -30q60 -164 216.5 -320.5t320.5 -216.5q6 -2 30 -11t33 -12.5 t29.5 -10t33 -9t30.5 -5t34 -2.5q57 0 130.5 34t94.5 80q22 53 22 101zM1536 1120v-960q0 -119 -84.5 -203.5t-203.5 -84.5h-960q-119 0 -203.5 84.5t-84.5 203.5v960q0 119 84.5 203.5t203.5 84.5h960q119 0 203.5 -84.5t84.5 -203.5z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1620 1128q-67 -98 -162 -167q1 -14 1 -42q0 -130 -38 -259.5t-115.5 -248.5t-184.5 -210.5t-258 -146t-323 -54.5q-271 0 -496 145q35 -4 78 -4q225 0 401 138q-105 2 -188 64.5t-114 159.5q33 -5 61 -5q43 0 85 11q-112 23 -185.5 111.5t-73.5 205.5v4q68 -38 146 -41 q-66 44 -105 115t-39 154q0 88 44 163q121 -149 294.5 -238.5t371.5 -99.5q-8 38 -8 74q0 134 94.5 228.5t228.5 94.5q140 0 236 -102q109 21 205 78q-37 -115 -142 -178q93 10 186 50z" />
+<glyph unicode="" horiz-adv-x="1024" d="M959 1524v-264h-157q-86 0 -116 -36t-30 -108v-189h293l-39 -296h-254v-759h-306v759h-255v296h255v218q0 186 104 288.5t277 102.5q147 0 228 -12z" />
+<glyph unicode="" d="M1536 640q0 -251 -146.5 -451.5t-378.5 -277.5q-27 -5 -39.5 7t-12.5 30v211q0 97 -52 142q57 6 102.5 18t94 39t81 66.5t53 105t20.5 150.5q0 121 -79 206q37 91 -8 204q-28 9 -81 -11t-92 -44l-38 -24q-93 26 -192 26t-192 -26q-16 11 -42.5 27t-83.5 38.5t-86 13.5 q-44 -113 -7 -204q-79 -85 -79 -206q0 -85 20.5 -150t52.5 -105t80.5 -67t94 -39t102.5 -18q-40 -36 -49 -103q-21 -10 -45 -15t-57 -5t-65.5 21.5t-55.5 62.5q-19 32 -48.5 52t-49.5 24l-20 3q-21 0 -29 -4.5t-5 -11.5t9 -14t13 -12l7 -5q22 -10 43.5 -38t31.5 -51l10 -23 q13 -38 44 -61.5t67 -30t69.5 -7t55.5 3.5l23 4q0 -38 0.5 -89t0.5 -54q0 -18 -13 -30t-40 -7q-232 77 -378.5 277.5t-146.5 451.5q0 209 103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" horiz-adv-x="1664" d="M1664 960v-256q0 -26 -19 -45t-45 -19h-64q-26 0 -45 19t-19 45v256q0 106 -75 181t-181 75t-181 -75t-75 -181v-192h96q40 0 68 -28t28 -68v-576q0 -40 -28 -68t-68 -28h-960q-40 0 -68 28t-28 68v576q0 40 28 68t68 28h672v192q0 185 131.5 316.5t316.5 131.5 t316.5 -131.5t131.5 -316.5z" />
+<glyph unicode="" horiz-adv-x="1920" d="M1760 1408q66 0 113 -47t47 -113v-1216q0 -66 -47 -113t-113 -47h-1600q-66 0 -113 47t-47 113v1216q0 66 47 113t113 47h1600zM160 1280q-13 0 -22.5 -9.5t-9.5 -22.5v-224h1664v224q0 13 -9.5 22.5t-22.5 9.5h-1600zM1760 0q13 0 22.5 9.5t9.5 22.5v608h-1664v-608 q0 -13 9.5 -22.5t22.5 -9.5h1600zM256 128v128h256v-128h-256zM640 128v128h384v-128h-384z" />
+<glyph unicode="" horiz-adv-x="1408" d="M384 192q0 -80 -56 -136t-136 -56t-136 56t-56 136t56 136t136 56t136 -56t56 -136zM896 69q2 -28 -17 -48q-18 -21 -47 -21h-135q-25 0 -43 16.5t-20 41.5q-22 229 -184.5 391.5t-391.5 184.5q-25 2 -41.5 20t-16.5 43v135q0 29 21 47q17 17 43 17h5q160 -13 306 -80.5 t259 -181.5q114 -113 181.5 -259t80.5 -306zM1408 67q2 -27 -18 -47q-18 -20 -46 -20h-143q-26 0 -44.5 17.5t-19.5 42.5q-12 215 -101 408.5t-231.5 336t-336 231.5t-408.5 102q-25 1 -42.5 19.5t-17.5 43.5v143q0 28 20 46q18 18 44 18h3q262 -13 501.5 -120t425.5 -294 q187 -186 294 -425.5t120 -501.5z" />
+<glyph unicode="" d="M1040 320q0 -33 -23.5 -56.5t-56.5 -23.5t-56.5 23.5t-23.5 56.5t23.5 56.5t56.5 23.5t56.5 -23.5t23.5 -56.5zM1296 320q0 -33 -23.5 -56.5t-56.5 -23.5t-56.5 23.5t-23.5 56.5t23.5 56.5t56.5 23.5t56.5 -23.5t23.5 -56.5zM1408 160v320q0 13 -9.5 22.5t-22.5 9.5 h-1216q-13 0 -22.5 -9.5t-9.5 -22.5v-320q0 -13 9.5 -22.5t22.5 -9.5h1216q13 0 22.5 9.5t9.5 22.5zM178 640h1180l-157 482q-4 13 -16 21.5t-26 8.5h-782q-14 0 -26 -8.5t-16 -21.5zM1536 480v-320q0 -66 -47 -113t-113 -47h-1216q-66 0 -113 47t-47 113v320q0 25 16 75 l197 606q17 53 63 86t101 33h782q55 0 101 -33t63 -86l197 -606q16 -50 16 -75z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1664 896q53 0 90.5 -37.5t37.5 -90.5t-37.5 -90.5t-90.5 -37.5v-384q0 -52 -38 -90t-90 -38q-417 347 -812 380q-58 -19 -91 -66t-31 -100.5t40 -92.5q-20 -33 -23 -65.5t6 -58t33.5 -55t48 -50t61.5 -50.5q-29 -58 -111.5 -83t-168.5 -11.5t-132 55.5q-7 23 -29.5 87.5 t-32 94.5t-23 89t-15 101t3.5 98.5t22 110.5h-122q-66 0 -113 47t-47 113v192q0 66 47 113t113 47h480q435 0 896 384q52 0 90 -38t38 -90v-384zM1536 292v954q-394 -302 -768 -343v-270q377 -42 768 -341z" />
+<glyph unicode="" horiz-adv-x="1792" d="M912 -160q0 16 -16 16q-59 0 -101.5 42.5t-42.5 101.5q0 16 -16 16t-16 -16q0 -73 51.5 -124.5t124.5 -51.5q16 0 16 16zM246 128h1300q-266 300 -266 832q0 51 -24 105t-69 103t-121.5 80.5t-169.5 31.5t-169.5 -31.5t-121.5 -80.5t-69 -103t-24 -105q0 -532 -266 -832z M1728 128q0 -52 -38 -90t-90 -38h-448q0 -106 -75 -181t-181 -75t-181 75t-75 181h-448q-52 0 -90 38t-38 90q50 42 91 88t85 119.5t74.5 158.5t50 206t19.5 260q0 152 117 282.5t307 158.5q-8 19 -8 39q0 40 28 68t68 28t68 -28t28 -68q0 -20 -8 -39q190 -28 307 -158.5 t117 -282.5q0 -139 19.5 -260t50 -206t74.5 -158.5t85 -119.5t91 -88z" />
+<glyph unicode="" d="M1376 640l138 -135q30 -28 20 -70q-12 -41 -52 -51l-188 -48l53 -186q12 -41 -19 -70q-29 -31 -70 -19l-186 53l-48 -188q-10 -40 -51 -52q-12 -2 -19 -2q-31 0 -51 22l-135 138l-135 -138q-28 -30 -70 -20q-41 11 -51 52l-48 188l-186 -53q-41 -12 -70 19q-31 29 -19 70 l53 186l-188 48q-40 10 -52 51q-10 42 20 70l138 135l-138 135q-30 28 -20 70q12 41 52 51l188 48l-53 186q-12 41 19 70q29 31 70 19l186 -53l48 188q10 41 51 51q41 12 70 -19l135 -139l135 139q29 30 70 19q41 -10 51 -51l48 -188l186 53q41 12 70 -19q31 -29 19 -70 l-53 -186l188 -48q40 -10 52 -51q10 -42 -20 -70z" />
+<glyph unicode="" horiz-adv-x="1792" d="M256 192q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1664 768q0 51 -39 89.5t-89 38.5h-576q0 20 15 48.5t33 55t33 68t15 84.5q0 67 -44.5 97.5t-115.5 30.5q-24 0 -90 -139q-24 -44 -37 -65q-40 -64 -112 -145q-71 -81 -101 -106 q-69 -57 -140 -57h-32v-640h32q72 0 167 -32t193.5 -64t179.5 -32q189 0 189 167q0 26 -5 56q30 16 47.5 52.5t17.5 73.5t-18 69q53 50 53 119q0 25 -10 55.5t-25 47.5h331q52 0 90 38t38 90zM1792 769q0 -105 -75.5 -181t-180.5 -76h-169q-4 -62 -37 -119q3 -21 3 -43 q0 -101 -60 -178q1 -139 -85 -219.5t-227 -80.5q-133 0 -322 69q-164 59 -223 59h-288q-53 0 -90.5 37.5t-37.5 90.5v640q0 53 37.5 90.5t90.5 37.5h288q10 0 21.5 4.5t23.5 14t22.5 18t24 22.5t20.5 21.5t19 21.5t14 17q65 74 100 129q13 21 33 62t37 72t40.5 63t55 49.5 t69.5 17.5q125 0 206.5 -67t81.5 -189q0 -68 -22 -128h374q104 0 180 -76t76 -179z" />
+<glyph unicode="" horiz-adv-x="1792" d="M1376 128h32v640h-32q-35 0 -67.5 12t-62.5 37t-50 46t-49 54q-2 3 -3.5 4.5t-4 4.5t-4.5 5q-72 81 -112 145q-14 22 -38 68q-1 3 -10.5 22.5t-18.5 36t-20 35.5t-21.5 30.5t-18.5 11.5q-71 0 -115.5 -30.5t-44.5 -97.5q0 -43 15 -84.5t33 -68t33 -55t15 -48.5h-576 q-50 0 -89 -38.5t-39 -89.5q0 -52 38 -90t90 -38h331q-15 -17 -25 -47.5t-10 -55.5q0 -69 53 -119q-18 -32 -18 -69t17.5 -73.5t47.5 -52.5q-4 -24 -4 -56q0 -85 48.5 -126t135.5 -41q84 0 183 32t194 64t167 32zM1664 192q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45 t45 -19t45 19t19 45zM1792 768v-640q0 -53 -37.5 -90.5t-90.5 -37.5h-288q-59 0 -223 -59q-190 -69 -317 -69q-142 0 -230 77.5t-87 217.5l1 5q-61 76 -61 178q0 22 3 43q-33 57 -37 119h-169q-105 0 -180.5 76t-75.5 181q0 103 76 179t180 76h374q-22 60 -22 128 q0 122 81.5 189t206.5 67q38 0 69.5 -17.5t55 -49.5t40.5 -63t37 -72t33 -62q35 -55 100 -129q2 -3 14 -17t19 -21.5t20.5 -21.5t24 -22.5t22.5 -18t23.5 -14t21.5 -4.5h288q53 0 90.5 -37.5t37.5 -90.5z" />
+<glyph unicode="" d="M1280 -64q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1408 700q0 189 -167 189q-26 0 -56 -5q-16 30 -52.5 47.5t-73.5 17.5t-69 -18q-50 53 -119 53q-25 0 -55.5 -10t-47.5 -25v331q0 52 -38 90t-90 38q-51 0 -89.5 -39t-38.5 -89v-576 q-20 0 -48.5 15t-55 33t-68 33t-84.5 15q-67 0 -97.5 -44.5t-30.5 -115.5q0 -24 139 -90q44 -24 65 -37q64 -40 145 -112q81 -71 106 -101q57 -69 57 -140v-32h640v32q0 72 32 167t64 193.5t32 179.5zM1536 705q0 -133 -69 -322q-59 -164 -59 -223v-288q0 -53 -37.5 -90.5 t-90.5 -37.5h-640q-53 0 -90.5 37.5t-37.5 90.5v288q0 10 -4.5 21.5t-14 23.5t-18 22.5t-22.5 24t-21.5 20.5t-21.5 19t-17 14q-74 65 -129 100q-21 13 -62 33t-72 37t-63 40.5t-49.5 55t-17.5 69.5q0 125 67 206.5t189 81.5q68 0 128 -22v374q0 104 76 180t179 76 q105 0 181 -75.5t76 -180.5v-169q62 -4 119 -37q21 3 43 3q101 0 178 -60q139 1 219.5 -85t80.5 -227z" />
+<glyph unicode="" d="M1408 576q0 84 -32 183t-64 194t-32 167v32h-640v-32q0 -35 -12 -67.5t-37 -62.5t-46 -50t-54 -49q-9 -8 -14 -12q-81 -72 -145 -112q-22 -14 -68 -38q-3 -1 -22.5 -10.5t-36 -18.5t-35.5 -20t-30.5 -21.5t-11.5 -18.5q0 -71 30.5 -115.5t97.5 -44.5q43 0 84.5 15t68 33 t55 33t48.5 15v-576q0 -50 38.5 -89t89.5 -39q52 0 90 38t38 90v331q46 -35 103 -35q69 0 119 53q32 -18 69 -18t73.5 17.5t52.5 47.5q24 -4 56 -4q85 0 126 48.5t41 135.5zM1280 1344q0 26 -19 45t-45 19t-45 -19t-19 -45t19 -45t45 -19t45 19t19 45zM1536 580 q0 -142 -77.5 -230t-217.5 -87l-5 1q-76 -61 -178 -61q-22 0 -43 3q-54 -30 -119 -37v-169q0 -105 -76 -180.5t-181 -75.5q-103 0 -179 76t-76 180v374q-54 -22 -128 -22q-121 0 -188.5 81.5t-67.5 206.5q0 38 17.5 69.5t49.5 55t63 40.5t72 37t62 33q55 35 129 100 q3 2 17 14t21.5 19t21.5 20.5t22.5 24t18 22.5t14 23.5t4.5 21.5v288q0 53 37.5 90.5t90.5 37.5h640q53 0 90.5 -37.5t37.5 -90.5v-288q0 -59 59 -223q69 -190 69 -317z" />
+<glyph unicode="" d="M1280 576v128q0 26 -19 45t-45 19h-502l189 189q19 19 19 45t-19 45l-91 91q-18 18 -45 18t-45 -18l-362 -362l-91 -91q-18 -18 -18 -45t18 -45l91 -91l362 -362q18 -18 45 -18t45 18l91 91q18 18 18 45t-18 45l-189 189h502q26 0 45 19t19 45zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" d="M1285 640q0 27 -18 45l-91 91l-362 362q-18 18 -45 18t-45 -18l-91 -91q-18 -18 -18 -45t18 -45l189 -189h-502q-26 0 -45 -19t-19 -45v-128q0 -26 19 -45t45 -19h502l-189 -189q-19 -19 -19 -45t19 -45l91 -91q18 -18 45 -18t45 18l362 362l91 91q18 18 18 45zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" d="M1284 641q0 27 -18 45l-362 362l-91 91q-18 18 -45 18t-45 -18l-91 -91l-362 -362q-18 -18 -18 -45t18 -45l91 -91q18 -18 45 -18t45 18l189 189v-502q0 -26 19 -45t45 -19h128q26 0 45 19t19 45v502l189 -189q19 -19 45 -19t45 19l91 91q18 18 18 45zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" d="M1284 639q0 27 -18 45l-91 91q-18 18 -45 18t-45 -18l-189 -189v502q0 26 -19 45t-45 19h-128q-26 0 -45 -19t-19 -45v-502l-189 189q-19 19 -45 19t-45 -19l-91 -91q-18 -18 -18 -45t18 -45l362 -362l91 -91q18 -18 45 -18t45 18l91 91l362 362q18 18 18 45zM1536 640 q0 -209 -103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103t385.5 -103t279.5 -279.5t103 -385.5z" />
+<glyph unicode="" d="M768 1408q209 0 385.5 -103t279.5 -279.5t103 -385.5t-103 -385.5t-279.5 -279.5t-385.5 -103t-385.5 103t-279.5 279.5t-103 385.5t103 385.5t279.5 279.5t385.5 103zM1042 887q-2 -1 -9.5 -9.5t-13.5 -9.5q2 0 4.5 5t5 11t3.5 7q6 7 22 15q14 6 52 12q34 8 51 -11 q-2 2 9.5 13t14.5 12q3 2 15 4.5t15 7.5l2 22q-12 -1 -17.5 7t-6.5 21q0 -2 -6 -8q0 7 -4.5 8t-11.5 -1t-9 -1q-10 3 -15 7.5t-8 16.5t-4 15q-2 5 -9.5 10.5t-9.5 10.5q-1 2 -2.5 5.5t-3 6.5t-4 5.5t-5.5 2.5t-7 -5t-7.5 -10t-4.5 -5q-3 2 -6 1.5t-4.5 -1t-4.5 -3t-5 -3.5 q-3 -2 -8.5 -3t-8.5 -2q15 5 -1 11q-10 4 -16 3q9 4 7.5 12t-8.5 14h5q-1 4 -8.5 8.5t-17.5 8.5t-13 6q-8 5 -34 9.5t-33 0.5q-5 -6 -4.5 -10.5t4 -14t3.5 -12.5q1 -6 -5.5 -13t-6.5 -12q0 -7 14 -15.5t10 -21.5q-3 -8 -16 -16t-16 -12q-5 -8 -1.5 -18.5t10.5 -16.5 q2 -2 1.5 -4t-3.5 -4.5t-5.5 -4t-6.5 -3.5l-3 -2q-11 -5 -20.5 6t-13.5 26q-7 25 -16 30q-23 8 -29 -1q-5 13 -41 26q-25 9 -58 4q6 1 0 15q-7 15 -19 12q3 6 4 17.5t1 13.5q3 13 12 23q1 1 7 8.5t9.5 13.5t0.5 6q35 -4 50 11q5 5 11.5 17
t10.5 17q9 6 14 5.5t14.5 -5.5 t14.5 -5q14 -1 15.5 11t-7.5 20q12 -1 3 17q-5 7 -8 9q-12 4 -27 -5q-8 -4 2 -8q-1 1 -9.5 -10.5t-16.5 -17.5t-16 5q-1 1 -5.5 13.5t-9.5 13.5q-8 0 -16 -15q3 8 -11 15t-24 8q19 12 -8 27q-7 4 -20.5 5t-19.5 -4q-5 -7 -5.5 -11.5t5 -8t10.5 -5.5t11.5 -4t8.5 -3 q14 -10 8 -14q-2 -1 -8.5 -3.5t-11.5 -4.5t-6 -4q-3 -4 0 -14t-2 -14q-5 5 -9 17.5t-7 16.5q7 -9 -25 -6l-10 1q-4 0 -16 -2t-20.5 -1t-13.5 8q-4 8 0 20q1 4 4 2q-4 3 -11 9.5t-10 8.5q-46 -15 -94 -41q6 -1 12 1q5 2 13 6.5t10 5.5q34 14 42 7l5 5q14 -16 20 -25 q-7 4 -30 1q-20 -6 -22 -12q7 -12 5 -18q-4 3 -11.5 10t-14.5 11t-15 5q-16 0 -22 -1q-146 -80 -235 -222q7 -7 12 -8q4 -1 5 -9t2.5 -11t11.5 3q9 -8 3 -19q1 1 44 -27q19 -17 21 -21q3 -11 -10 -18q-1 2 -9 9t-9 4q-3 -5 0.5 -18.5t10.5 -12.5q-7 0 -9.5 -16t-2.5 -35.5 t-1 -23.5l2 -1q-3 -12 5.5 -34.5t21.5 -19.5q-13 -3 20 -43q6 -8 8 -9q3 -2 12 -7.5t15 -10t10 -10.5q4 -5 10 -22.5t14 -23.5q-2 -6 9.5 -20t10.5 -23q-1 0 -2.5 -1t-2.5 -1q3 -7 15.5 -14t15.5 -13q1 -3 2 -10t3 -11t8 -2q2 20 -24 62q-1
5 25 -17 29q-3 5 -5.5 15.5 t-4.5 14.5q2 0 6 -1.5t8.5 -3.5t7.5 -4t2 -3q-3 -7 2 -17.5t12 -18.5t17 -19t12 -13q6 -6 14 -19.5t0 -13.5q9 0 20 -10t17 -20q5 -8 8 -26t5 -24q2 -7 8.5 -13.5t12.5 -9.5l16 -8t13 -7q5 -2 18.5 -10.5t21.5 -11.5q10 -4 16 -4t14.5 2.5t13.5 3.5q15 2 29 -15t21 -21 q36 -19 55 -11q-2 -1 0.5 -7.5t8 -15.5t9 -14.5t5.5 -8.5q5 -6 18 -15t18 -15q6 4 7 9q-3 -8 7 -20t18 -10q14 3 14 32q-31 -15 -49 18q0 1 -2.5 5.5t-4 8.5t-2.5 8.5t0 7.5t5 3q9 0 10 3.5t-2 12.5t-4 13q-1 8 -11 20t-12 15q-5 -9 -16 -8t-16 9q0 -1 -1.5 -5.5t-1.5 -6.5 q-13 0 -15 1q1 3 2.5 17.5t3.5 22.5q1 4 5.5 12t7.5 14.5t4 12.5t-4.5 9.5t-17.5 2.5q-19 -1 -26 -20q-1 -3 -3 -10.5t-5 -11.5t-9 -7q-7 -3 -24 -2t-24 5q-13 8 -22.5 29t-9.5 37q0 10 2.5 26.5t3 25t-5.5 24.5q3 2 9 9.5t10 10.5q2 1 4.5 1.5t4.5 0t4 1.5t3 6q-1 1 -4 3 q-3 3 -4 3q7 -3 28.5 1.5t27.5 -1.5q15 -11 22 2q0 1 -2.5 9.5t-0.5 13.5q5 -27 29 -9q3 -3 15.5 -5t17.5 -5q3 -2 7 -5.5t5.5 -4.5t5 0.5t8.5 6.5q10 -14 12 -24q11 -40 19 -44q7 -3 11 -2t4.5 9.5t0 14t-1.5 12.5l-1 8v18l-1 8q
-15 3 -18.5 12t1.5 18.5t15 18.5q1 1 8 3.5 t15.5 6.5t12.5 8q21 19 15 35q7 0 11 9q-1 0 -5 3t-7.5 5t-4.5 2q9 5 2 16q5 3 7.5 11t7.5 10q9 -12 21 -2q7 8 1 16q5 7 20.5 10.5t18.5 9.5q7 -2 8 2t1 12t3 12q4 5 15 9t13 5l17 11q3 4 0 4q18 -2 31 11q10 11 -6 20q3 6 -3 9.5t-15 5.5q3 1 11.5 0.5t10.5 1.5 q15 10 -7 16q-17 5 -43 -12zM879 10q206 36 351 189q-3 3 -12.5 4.5t-12.5 3.5q-18 7 -24 8q1 7 -2.5 13t-8 9t-12.5 8t-11 7q-2 2 -7 6t-7 5.5t-7.5 4.5t-8.5 2t-10 -1l-3 -1q-3 -1 -5.5 -2.5t-5.5 -3t-4 -3t0 -2.5q-21 17 -36 22q-5 1 -11 5.5t-10.5 7t-10 1.5t-11.5 -7 q-5 -5 -6 -15t-2 -13q-7 5 0 17.5t2 18.5q-3 6 -10.5 4.5t-12 -4.5t-11.5 -8.5t-9 -6.5t-8.5 -5.5t-8.5 -7.5q-3 -4 -6 -12t-5 -11q-2 4 -11.5 6.5t-9.5 5.5q2 -10 4 -35t5 -38q7
<TRUNCATED>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/fonts/fontawesome-webfont.ttf
----------------------------------------------------------------------
diff --git a/_static/fonts/fontawesome-webfont.ttf b/_static/fonts/fontawesome-webfont.ttf
new file mode 100644
index 0000000..96a3639
Binary files /dev/null and b/_static/fonts/fontawesome-webfont.ttf differ
[25/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/hive_hooks.html
----------------------------------------------------------------------
diff --git a/_modules/hive_hooks.html b/_modules/hive_hooks.html
new file mode 100644
index 0000000..15c45e2
--- /dev/null
+++ b/_modules/hive_hooks.html
@@ -0,0 +1,743 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>hive_hooks — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>hive_hooks</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for hive_hooks</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="c1">#</span>
+
+<span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">print_function</span>
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">zip</span>
+<span class="kn">from</span> <span class="nn">past.builtins</span> <span class="kn">import</span> <span class="nb">basestring</span>
+<span class="kn">import</span> <span class="nn">unicodecsv</span> <span class="kn">as</span> <span class="nn">csv</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">import</span> <span class="nn">re</span>
+<span class="kn">import</span> <span class="nn">subprocess</span>
+<span class="kn">from</span> <span class="nn">tempfile</span> <span class="kn">import</span> <span class="n">NamedTemporaryFile</span>
+
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="kn">import</span> <span class="n">BaseHook</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.file</span> <span class="kn">import</span> <span class="n">TemporaryDirectory</span>
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="kn">import</span> <span class="n">configuration</span>
+<span class="kn">import</span> <span class="nn">airflow.security.utils</span> <span class="kn">as</span> <span class="nn">utils</span>
+
+
+<div class="viewcode-block" id="HiveCliHook"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveCliHook">[docs]</a><span class="k">class</span> <span class="nc">HiveCliHook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
+
+ <span class="sd">"""Simple wrapper around the hive CLI.</span>
+
+<span class="sd"> It also supports the ``beeline``</span>
+<span class="sd"> a lighter CLI that runs JDBC and is replacing the heavier</span>
+<span class="sd"> traditional CLI. To enable ``beeline``, set the use_beeline param in the</span>
+<span class="sd"> extra field of your connection as in ``{ "use_beeline": true }``</span>
+
+<span class="sd"> Note that you can also set default hive CLI parameters using the</span>
+<span class="sd"> ``hive_cli_params`` to be used in your connection as in</span>
+<span class="sd"> ``{"hive_cli_params": "-hiveconf mapred.job.tracker=some.jobtracker:444"}``</span>
+
+<span class="sd"> The extra connection parameter ``auth`` gets passed as in the ``jdbc``</span>
+<span class="sd"> connection string as is.</span>
+<span class="sd"> """</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">hive_cli_conn_id</span><span class="o">=</span><span class="s2">"hive_cli_default"</span><span class="p">,</span>
+ <span class="n">run_as</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="n">hive_cli_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_params</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'hive_cli_params'</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">use_beeline</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'use_beeline'</span><span class="p">,</span> <span class="bp">False</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">auth</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'auth'</span><span class="p">,</span> <span class="s1">'noSasl'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">conn</span> <span class="o">=</span> <span class="n">conn</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">run_as</span> <span class="o">=</span> <span class="n">run_as</span>
+
+<div class="viewcode-block" id="HiveCliHook.run_cli"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveCliHook.run_cli">[docs]</a> <span class="k">def</span> <span class="nf">run_cli</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Run an hql statement using the hive cli</span>
+
+<span class="sd"> >>> hh = HiveCliHook()</span>
+<span class="sd"> >>> result = hh.run_cli("USE airflow;")</span>
+<span class="sd"> >>> ("OK" in result)</span>
+<span class="sd"> True</span>
+<span class="sd"> """</span>
+ <span class="n">conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">conn</span>
+ <span class="n">schema</span> <span class="o">=</span> <span class="n">schema</span> <span class="ow">or</span> <span class="n">conn</span><span class="o">.</span><span class="n">schema</span>
+ <span class="k">if</span> <span class="n">schema</span><span class="p">:</span>
+ <span class="n">hql</span> <span class="o">=</span> <span class="s2">"USE {schema};</span><span class="se">\n</span><span class="s2">{hql}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
+
+ <span class="k">with</span> <span class="n">TemporaryDirectory</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="s1">'airflow_hiveop_'</span><span class="p">)</span> <span class="k">as</span> <span class="n">tmp_dir</span><span class="p">:</span>
+ <span class="k">with</span> <span class="n">NamedTemporaryFile</span><span class="p">(</span><span class="nb">dir</span><span class="o">=</span><span class="n">tmp_dir</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">hql</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'UTF-8'</span><span class="p">))</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
+ <span class="n">fname</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">name</span>
+ <span class="n">hive_bin</span> <span class="o">=</span> <span class="s1">'hive'</span>
+ <span class="n">cmd_extra</span> <span class="o">=</span> <span class="p">[]</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">use_beeline</span><span class="p">:</span>
+ <span class="n">hive_bin</span> <span class="o">=</span> <span class="s1">'beeline'</span>
+ <span class="n">jdbc_url</span> <span class="o">=</span> <span class="s2">"jdbc:hive2://{conn.host}:{conn.port}/{conn.schema}"</span>
+ <span class="k">if</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'security'</span><span class="p">)</span> <span class="o">==</span> <span class="s1">'kerberos'</span><span class="p">:</span>
+ <span class="n">template</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span>
+ <span class="s1">'principal'</span><span class="p">,</span> <span class="s2">"hive/_HOST@EXAMPLE.COM"</span><span class="p">)</span>
+ <span class="k">if</span> <span class="s2">"_HOST"</span> <span class="ow">in</span> <span class="n">template</span><span class="p">:</span>
+ <span class="n">template</span> <span class="o">=</span> <span class="n">utils</span><span class="o">.</span><span class="n">replace_hostname_pattern</span><span class="p">(</span>
+ <span class="n">utils</span><span class="o">.</span><span class="n">get_components</span><span class="p">(</span><span class="n">template</span><span class="p">))</span>
+
+ <span class="n">proxy_user</span> <span class="o">=</span> <span class="s2">""</span> <span class="c1"># noqa</span>
+ <span class="k">if</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'proxy_user'</span><span class="p">)</span> <span class="o">==</span> <span class="s2">"login"</span> <span class="ow">and</span> <span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">:</span>
+ <span class="n">proxy_user</span> <span class="o">=</span> <span class="s2">"hive.server2.proxy.user={0}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="n">conn</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'proxy_user'</span><span class="p">)</span> <span class="o">==</span> <span class="s2">"owner"</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_as</span><span class="p">:</span>
+ <span class="n">proxy_user</span> <span class="o">=</span> <span class="s2">"hive.server2.proxy.user={0}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">run_as</span><span class="p">)</span>
+
+ <span class="n">jdbc_url</span> <span class="o">+=</span> <span class="s2">";principal={template};{proxy_user}"</span>
+ <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">auth</span><span class="p">:</span>
+ <span class="n">jdbc_url</span> <span class="o">+=</span> <span class="s2">";auth="</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">auth</span>
+
+ <span class="n">jdbc_url</span> <span class="o">=</span> <span class="n">jdbc_url</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
+
+ <span class="n">cmd_extra</span> <span class="o">+=</span> <span class="p">[</span><span class="s1">'-u'</span><span class="p">,</span> <span class="n">jdbc_url</span><span class="p">]</span>
+ <span class="k">if</span> <span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">:</span>
+ <span class="n">cmd_extra</span> <span class="o">+=</span> <span class="p">[</span><span class="s1">'-n'</span><span class="p">,</span> <span class="n">conn</span><span class="o">.</span><span class="n">login</span><span class="p">]</span>
+ <span class="k">if</span> <span class="n">conn</span><span class="o">.</span><span class="n">password</span><span class="p">:</span>
+ <span class="n">cmd_extra</span> <span class="o">+=</span> <span class="p">[</span><span class="s1">'-p'</span><span class="p">,</span> <span class="n">conn</span><span class="o">.</span><span class="n">password</span><span class="p">]</span>
+
+ <span class="n">hive_cmd</span> <span class="o">=</span> <span class="p">[</span><span class="n">hive_bin</span><span class="p">,</span> <span class="s1">'-f'</span><span class="p">,</span> <span class="n">fname</span><span class="p">]</span> <span class="o">+</span> <span class="n">cmd_extra</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_params</span><span class="p">:</span>
+ <span class="n">hive_params_list</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">hive_cli_params</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
+ <span class="n">hive_cmd</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">hive_params_list</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">verbose</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">" "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">hive_cmd</span><span class="p">))</span>
+ <span class="n">sp</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">Popen</span><span class="p">(</span>
+ <span class="n">hive_cmd</span><span class="p">,</span>
+ <span class="n">stdout</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">PIPE</span><span class="p">,</span>
+ <span class="n">stderr</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">STDOUT</span><span class="p">,</span>
+ <span class="n">cwd</span><span class="o">=</span><span class="n">tmp_dir</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sp</span> <span class="o">=</span> <span class="n">sp</span>
+ <span class="n">stdout</span> <span class="o">=</span> <span class="s1">''</span>
+ <span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
+ <span class="n">line</span> <span class="o">=</span> <span class="n">sp</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">line</span><span class="p">:</span>
+ <span class="k">break</span>
+ <span class="n">stdout</span> <span class="o">+=</span> <span class="n">line</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'UTF-8'</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">verbose</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s1">'UTF-8'</span><span class="p">)</span><span class="o">.</span><span class="n">strip</span><span class="p">())</span>
+ <span class="n">sp</span><span class="o">.</span><span class="n">wait</span><span class="p">()</span>
+
+ <span class="k">if</span> <span class="n">sp</span><span class="o">.</span><span class="n">returncode</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="n">stdout</span><span class="p">)</span>
+
+ <span class="k">return</span> <span class="n">stdout</span></div>
+
+<div class="viewcode-block" id="HiveCliHook.test_hql"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveCliHook.test_hql">[docs]</a> <span class="k">def</span> <span class="nf">test_hql</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Test an hql statement using the hive cli and EXPLAIN</span>
+
+<span class="sd"> """</span>
+ <span class="n">create</span><span class="p">,</span> <span class="n">insert</span><span class="p">,</span> <span class="n">other</span> <span class="o">=</span> <span class="p">[],</span> <span class="p">[],</span> <span class="p">[]</span>
+ <span class="k">for</span> <span class="n">query</span> <span class="ow">in</span> <span class="n">hql</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">';'</span><span class="p">):</span> <span class="c1"># naive</span>
+ <span class="n">query_original</span> <span class="o">=</span> <span class="n">query</span>
+ <span class="n">query</span> <span class="o">=</span> <span class="n">query</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
+
+ <span class="k">if</span> <span class="n">query</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'create table'</span><span class="p">):</span>
+ <span class="n">create</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">query_original</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="n">query</span><span class="o">.</span><span class="n">startswith</span><span class="p">((</span><span class="s1">'set '</span><span class="p">,</span>
+ <span class="s1">'add jar '</span><span class="p">,</span>
+ <span class="s1">'create temporary function'</span><span class="p">)):</span>
+ <span class="n">other</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">query_original</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="n">query</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'insert'</span><span class="p">):</span>
+ <span class="n">insert</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">query_original</span><span class="p">)</span>
+ <span class="n">other</span> <span class="o">=</span> <span class="s1">';'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">other</span><span class="p">)</span>
+ <span class="k">for</span> <span class="n">query_set</span> <span class="ow">in</span> <span class="p">[</span><span class="n">create</span><span class="p">,</span> <span class="n">insert</span><span class="p">]:</span>
+ <span class="k">for</span> <span class="n">query</span> <span class="ow">in</span> <span class="n">query_set</span><span class="p">:</span>
+
+ <span class="n">query_preview</span> <span class="o">=</span> <span class="s1">' '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">query</span><span class="o">.</span><span class="n">split</span><span class="p">())[:</span><span class="mi">50</span><span class="p">]</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Testing HQL [{0} (...)]"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">query_preview</span><span class="p">))</span>
+ <span class="k">if</span> <span class="n">query_set</span> <span class="o">==</span> <span class="n">insert</span><span class="p">:</span>
+ <span class="n">query</span> <span class="o">=</span> <span class="n">other</span> <span class="o">+</span> <span class="s1">'; explain '</span> <span class="o">+</span> <span class="n">query</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">query</span> <span class="o">=</span> <span class="s1">'explain '</span> <span class="o">+</span> <span class="n">query</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="n">query</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
+ <span class="k">except</span> <span class="n">AirflowException</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="n">message</span> <span class="o">=</span> <span class="n">e</span><span class="o">.</span><span class="n">args</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)[</span><span class="o">-</span><span class="mi">2</span><span class="p">]</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
+ <span class="n">error_loc</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="s1">'(\d+):(\d+)'</span><span class="p">,</span> <span class="n">message</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">error_loc</span> <span class="ow">and</span> <span class="n">error_loc</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">isdigit</span><span class="p">():</span>
+ <span class="n">l</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">error_loc</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">))</span>
+ <span class="n">begin</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="n">l</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="mi">0</span><span class="p">)</span>
+ <span class="n">end</span> <span class="o">=</span> <span class="nb">min</span><span class="p">(</span><span class="n">l</span><span class="o">+</span><span class="mi">3</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">query</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)))</span>
+ <span class="n">context</span> <span class="o">=</span> <span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">query</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)[</span><span class="n">begin</span><span class="p">:</span><span class="n">end</span><span class="p">])</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Context :</span><span class="se">\n</span><span class="s2"> {0}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">context</span><span class="p">))</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"SUCCESS"</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="HiveCliHook.load_file"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveCliHook.load_file">[docs]</a> <span class="k">def</span> <span class="nf">load_file</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">filepath</span><span class="p">,</span>
+ <span class="n">table</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="s2">","</span><span class="p">,</span>
+ <span class="n">field_dict</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">create</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
+ <span class="n">overwrite</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
+ <span class="n">partition</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">recreate</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Loads a local file into Hive</span>
+
+<span class="sd"> Note that the table generated in Hive uses ``STORED AS textfile``</span>
+<span class="sd"> which isn't the most efficient serialization format. If a</span>
+<span class="sd"> large amount of data is loaded and/or if the tables gets</span>
+<span class="sd"> queried considerably, you may want to use this operator only to</span>
+<span class="sd"> stage the data into a temporary table before loading it into its</span>
+<span class="sd"> final destination using a ``HiveOperator``.</span>
+
+<span class="sd"> :param table: target Hive table, use dot notation to target a</span>
+<span class="sd"> specific database</span>
+<span class="sd"> :type table: str</span>
+<span class="sd"> :param create: whether to create the table if it doesn't exist</span>
+<span class="sd"> :type create: bool</span>
+<span class="sd"> :param recreate: whether to drop and recreate the table at every</span>
+<span class="sd"> execution</span>
+<span class="sd"> :type recreate: bool</span>
+<span class="sd"> :param partition: target partition as a dict of partition columns</span>
+<span class="sd"> and values</span>
+<span class="sd"> :type partition: dict</span>
+<span class="sd"> :param delimiter: field delimiter in the file</span>
+<span class="sd"> :type delimiter: str</span>
+<span class="sd"> """</span>
+ <span class="n">hql</span> <span class="o">=</span> <span class="s1">''</span>
+ <span class="k">if</span> <span class="n">recreate</span><span class="p">:</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"DROP TABLE IF EXISTS {table};</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="k">if</span> <span class="n">create</span> <span class="ow">or</span> <span class="n">recreate</span><span class="p">:</span>
+ <span class="n">fields</span> <span class="o">=</span> <span class="s2">",</span><span class="se">\n</span><span class="s2"> "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
+ <span class="p">[</span><span class="n">k</span> <span class="o">+</span> <span class="s1">' '</span> <span class="o">+</span> <span class="n">v</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">field_dict</span><span class="o">.</span><span class="n">items</span><span class="p">()])</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"CREATE TABLE IF NOT EXISTS {table} (</span><span class="se">\n</span><span class="s2">{fields})</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="k">if</span> <span class="n">partition</span><span class="p">:</span>
+ <span class="n">pfields</span> <span class="o">=</span> <span class="s2">",</span><span class="se">\n</span><span class="s2"> "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
+ <span class="p">[</span><span class="n">p</span> <span class="o">+</span> <span class="s2">" STRING"</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">partition</span><span class="p">])</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"PARTITIONED BY ({pfields})</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"ROW FORMAT DELIMITED</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"FIELDS TERMINATED BY '{delimiter}'</span><span class="se">\n</span><span class="s2">"</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"STORED AS textfile;"</span>
+ <span class="n">hql</span> <span class="o">=</span> <span class="n">hql</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span>
+ <span class="n">hql</span> <span class="o">=</span> <span class="s2">"LOAD DATA LOCAL INPATH '{filepath}' "</span>
+ <span class="k">if</span> <span class="n">overwrite</span><span class="p">:</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"OVERWRITE "</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"INTO TABLE {table} "</span>
+ <span class="k">if</span> <span class="n">partition</span><span class="p">:</span>
+ <span class="n">pvals</span> <span class="o">=</span> <span class="s2">", "</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
+ <span class="p">[</span><span class="s2">"{0}='{1}'"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">partition</span><span class="o">.</span><span class="n">items</span><span class="p">()])</span>
+ <span class="n">hql</span> <span class="o">+=</span> <span class="s2">"PARTITION ({pvals});"</span>
+ <span class="n">hql</span> <span class="o">=</span> <span class="n">hql</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">())</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">run_cli</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span></div>
+
+ <span class="k">def</span> <span class="nf">kill</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s1">'sp'</span><span class="p">):</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">sp</span><span class="o">.</span><span class="n">poll</span><span class="p">()</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
+ <span class="k">print</span><span class="p">(</span><span class="s2">"Killing the Hive job"</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sp</span><span class="o">.</span><span class="n">kill</span><span class="p">()</span></div>
+
+
+<div class="viewcode-block" id="HiveMetastoreHook"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveMetastoreHook">[docs]</a><span class="k">class</span> <span class="nc">HiveMetastoreHook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
+
+ <span class="sd">""" Wrapper to interact with the Hive Metastore"""</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">metastore_conn_id</span><span class="o">=</span><span class="s1">'metastore_default'</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metastore_conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="n">metastore_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_metastore_client</span><span class="p">()</span>
+
+ <span class="k">def</span> <span class="nf">__getstate__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="c1"># This is for pickling to work despite the thirft hive client not</span>
+ <span class="c1"># being pickable</span>
+ <span class="n">d</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">__dict__</span><span class="p">)</span>
+ <span class="k">del</span> <span class="n">d</span><span class="p">[</span><span class="s1">'metastore'</span><span class="p">]</span>
+ <span class="k">return</span> <span class="n">d</span>
+
+ <span class="k">def</span> <span class="nf">__setstate__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">d</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">__dict__</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">d</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">__dict__</span><span class="p">[</span><span class="s1">'metastore'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_metastore_client</span><span class="p">()</span>
+
+<div class="viewcode-block" id="HiveMetastoreHook.get_metastore_client"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveMetastoreHook.get_metastore_client">[docs]</a> <span class="k">def</span> <span class="nf">get_metastore_client</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a Hive thrift client.</span>
+<span class="sd"> """</span>
+ <span class="kn">from</span> <span class="nn">thrift.transport</span> <span class="kn">import</span> <span class="n">TSocket</span><span class="p">,</span> <span class="n">TTransport</span>
+ <span class="kn">from</span> <span class="nn">thrift.protocol</span> <span class="kn">import</span> <span class="n">TBinaryProtocol</span>
+ <span class="kn">from</span> <span class="nn">hive_service</span> <span class="kn">import</span> <span class="n">ThriftHive</span>
+ <span class="n">ms</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">metastore_conn</span>
+ <span class="n">auth_mechanism</span> <span class="o">=</span> <span class="n">ms</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'authMechanism'</span><span class="p">,</span> <span class="s1">'NOSASL'</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'security'</span><span class="p">)</span> <span class="o">==</span> <span class="s1">'kerberos'</span><span class="p">:</span>
+ <span class="n">auth_mechanism</span> <span class="o">=</span> <span class="n">ms</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'authMechanism'</span><span class="p">,</span> <span class="s1">'GSSAPI'</span><span class="p">)</span>
+ <span class="n">kerberos_service_name</span> <span class="o">=</span> <span class="n">ms</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'kerberos_service_name'</span><span class="p">,</span> <span class="s1">'hive'</span><span class="p">)</span>
+
+ <span class="n">socket</span> <span class="o">=</span> <span class="n">TSocket</span><span class="o">.</span><span class="n">TSocket</span><span class="p">(</span><span class="n">ms</span><span class="o">.</span><span class="n">host</span><span class="p">,</span> <span class="n">ms</span><span class="o">.</span><span class="n">port</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'security'</span><span class="p">)</span> <span class="o">==</span> <span class="s1">'kerberos'</span> <span class="ow">and</span> <span class="n">auth_mechanism</span> <span class="o">==</span> <span class="s1">'GSSAPI'</span><span class="p">:</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="kn">import</span> <span class="nn">saslwrapper</span> <span class="kn">as</span> <span class="nn">sasl</span>
+ <span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
+ <span class="kn">import</span> <span class="nn">sasl</span>
+
+ <span class="k">def</span> <span class="nf">sasl_factory</span><span class="p">():</span>
+ <span class="n">sasl_client</span> <span class="o">=</span> <span class="n">sasl</span><span class="o">.</span><span class="n">Client</span><span class="p">()</span>
+ <span class="n">sasl_client</span><span class="o">.</span><span class="n">setAttr</span><span class="p">(</span><span class="s2">"host"</span><span class="p">,</span> <span class="n">ms</span><span class="o">.</span><span class="n">host</span><span class="p">)</span>
+ <span class="n">sasl_client</span><span class="o">.</span><span class="n">setAttr</span><span class="p">(</span><span class="s2">"service"</span><span class="p">,</span> <span class="n">kerberos_service_name</span><span class="p">)</span>
+ <span class="n">sasl_client</span><span class="o">.</span><span class="n">init</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">sasl_client</span>
+
+ <span class="kn">from</span> <span class="nn">thrift_sasl</span> <span class="kn">import</span> <span class="n">TSaslClientTransport</span>
+ <span class="n">transport</span> <span class="o">=</span> <span class="n">TSaslClientTransport</span><span class="p">(</span><span class="n">sasl_factory</span><span class="p">,</span> <span class="s2">"GSSAPI"</span><span class="p">,</span> <span class="n">socket</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">transport</span> <span class="o">=</span> <span class="n">TTransport</span><span class="o">.</span><span class="n">TBufferedTransport</span><span class="p">(</span><span class="n">socket</span><span class="p">)</span>
+
+ <span class="n">protocol</span> <span class="o">=</span> <span class="n">TBinaryProtocol</span><span class="o">.</span><span class="n">TBinaryProtocol</span><span class="p">(</span><span class="n">transport</span><span class="p">)</span>
+
+ <span class="k">return</span> <span class="n">ThriftHive</span><span class="o">.</span><span class="n">Client</span><span class="p">(</span><span class="n">protocol</span><span class="p">)</span></div>
+
+ <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span>
+
+<div class="viewcode-block" id="HiveMetastoreHook.check_for_partition"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveMetastoreHook.check_for_partition">[docs]</a> <span class="k">def</span> <span class="nf">check_for_partition</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">schema</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">partition</span><span class="p">):</span>
+ <span class="sd">"""Checks whether a partition exists</span>
+
+<span class="sd"> >>> hh = HiveMetastoreHook()</span>
+<span class="sd"> >>> t = 'static_babynames_partitioned'</span>
+<span class="sd"> >>> hh.check_for_partition('airflow', t, "ds='2015-01-01'")</span>
+<span class="sd"> True</span>
+<span class="sd"> """</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">_oprot</span><span class="o">.</span><span class="n">trans</span><span class="o">.</span><span class="n">open</span><span class="p">()</span>
+ <span class="n">partitions</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">get_partitions_by_filter</span><span class="p">(</span>
+ <span class="n">schema</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">partition</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">_oprot</span><span class="o">.</span><span class="n">trans</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">partitions</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">True</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">False</span></div>
+
+<div class="viewcode-block" id="HiveMetastoreHook.get_table"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveMetastoreHook.get_table">[docs]</a> <span class="k">def</span> <span class="nf">get_table</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">table_name</span><span class="p">,</span> <span class="n">db</span><span class="o">=</span><span class="s1">'default'</span><span class="p">):</span>
+ <span class="sd">"""Get a metastore table object</span>
+
+<span class="sd"> >>> hh = HiveMetastoreHook()</span>
+<span class="sd"> >>> t = hh.get_table(db='airflow', table_name='static_babynames')</span>
+<span class="sd"> >>> t.tableName</span>
+<span class="sd"> 'static_babynames'</span>
+<span class="sd"> >>> [col.name for col in t.sd.cols]</span>
+<span class="sd"> ['state', 'year', 'name', 'gender', 'num']</span>
+<span class="sd"> """</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">_oprot</span><span class="o">.</span><span class="n">trans</span><span class="o">.</span><span class="n">open</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">db</span> <span class="o">==</span> <span class="s1">'default'</span> <span class="ow">and</span> <span class="s1">'.'</span> <span class="ow">in</span> <span class="n">table_name</span><span class="p">:</span>
+ <span class="n">db</span><span class="p">,</span> <span class="n">table_name</span> <span class="o">=</span> <span class="n">table_name</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)[:</span><span class="mi">2</span><span class="p">]</span>
+ <span class="n">table</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">get_table</span><span class="p">(</span><span class="n">dbname</span><span class="o">=</span><span class="n">db</span><span class="p">,</span> <span class="n">tbl_name</span><span class="o">=</span><span class="n">table_name</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">_oprot</span><span class="o">.</span><span class="n">trans</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">table</span></div>
+
+<div class="viewcode-block" id="HiveMetastoreHook.get_tables"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveMetastoreHook.get_tables">[docs]</a> <span class="k">def</span> <span class="nf">get_tables</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">db</span><span class="p">,</span> <span class="n">pattern</span><span class="o">=</span><span class="s1">'*'</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Get a metastore table object</span>
+<span class="sd"> """</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">_oprot</span><span class="o">.</span><span class="n">trans</span><span class="o">.</span><span class="n">open</span><span class="p">()</span>
+ <span class="n">tables</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">get_tables</span><span class="p">(</span><span class="n">db_name</span><span class="o">=</span><span class="n">db</span><span class="p">,</span> <span class="n">pattern</span><span class="o">=</span><span class="n">pattern</span><span class="p">)</span>
+ <span class="n">objs</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">get_table_objects_by_name</span><span class="p">(</span><span class="n">db</span><span class="p">,</span> <span class="n">tables</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">_oprot</span><span class="o">.</span><span class="n">trans</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">objs</span></div>
+
+<div class="viewcode-block" id="HiveMetastoreHook.get_databases"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveMetastoreHook.get_databases">[docs]</a> <span class="k">def</span> <span class="nf">get_databases</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">pattern</span><span class="o">=</span><span class="s1">'*'</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Get a metastore table object</span>
+<span class="sd"> """</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">_oprot</span><span class="o">.</span><span class="n">trans</span><span class="o">.</span><span class="n">open</span><span class="p">()</span>
+ <span class="n">dbs</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">get_databases</span><span class="p">(</span><span class="n">pattern</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">_oprot</span><span class="o">.</span><span class="n">trans</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">dbs</span></div>
+
+<div class="viewcode-block" id="HiveMetastoreHook.get_partitions"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveMetastoreHook.get_partitions">[docs]</a> <span class="k">def</span> <span class="nf">get_partitions</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">schema</span><span class="p">,</span> <span class="n">table_name</span><span class="p">,</span> <span class="nb">filter</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a list of all partitions in a table. Works only</span>
+<span class="sd"> for tables with less than 32767 (java short max val).</span>
+<span class="sd"> For subpartitioned table, the number might easily exceed this.</span>
+
+<span class="sd"> >>> hh = HiveMetastoreHook()</span>
+<span class="sd"> >>> t = 'static_babynames_partitioned'</span>
+<span class="sd"> >>> parts = hh.get_partitions(schema='airflow', table_name=t)</span>
+<span class="sd"> >>> len(parts)</span>
+<span class="sd"> 1</span>
+<span class="sd"> >>> parts</span>
+<span class="sd"> [{'ds': '2015-01-01'}]</span>
+<span class="sd"> """</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">_oprot</span><span class="o">.</span><span class="n">trans</span><span class="o">.</span><span class="n">open</span><span class="p">()</span>
+ <span class="n">table</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">get_table</span><span class="p">(</span><span class="n">dbname</span><span class="o">=</span><span class="n">schema</span><span class="p">,</span> <span class="n">tbl_name</span><span class="o">=</span><span class="n">table_name</span><span class="p">)</span>
+ <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">table</span><span class="o">.</span><span class="n">partitionKeys</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"The table isn't partitioned"</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">if</span> <span class="nb">filter</span><span class="p">:</span>
+ <span class="n">parts</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">get_partitions_by_filter</span><span class="p">(</span>
+ <span class="n">db_name</span><span class="o">=</span><span class="n">schema</span><span class="p">,</span> <span class="n">tbl_name</span><span class="o">=</span><span class="n">table_name</span><span class="p">,</span>
+ <span class="nb">filter</span><span class="o">=</span><span class="nb">filter</span><span class="p">,</span> <span class="n">max_parts</span><span class="o">=</span><span class="mi">32767</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">parts</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">get_partitions</span><span class="p">(</span>
+ <span class="n">db_name</span><span class="o">=</span><span class="n">schema</span><span class="p">,</span> <span class="n">tbl_name</span><span class="o">=</span><span class="n">table_name</span><span class="p">,</span> <span class="n">max_parts</span><span class="o">=</span><span class="mi">32767</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">metastore</span><span class="o">.</span><span class="n">_oprot</span><span class="o">.</span><span class="n">trans</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="n">pnames</span> <span class="o">=</span> <span class="p">[</span><span class="n">p</span><span class="o">.</span><span class="n">name</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">table</span><span class="o">.</span><span class="n">partitionKeys</span><span class="p">]</span>
+ <span class="k">return</span> <span class="p">[</span><span class="nb">dict</span><span class="p">(</span><span class="nb">zip</span><span class="p">(</span><span class="n">pnames</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">values</span><span class="p">))</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">parts</span><span class="p">]</span></div>
+
+<div class="viewcode-block" id="HiveMetastoreHook.max_partition"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveMetastoreHook.max_partition">[docs]</a> <span class="k">def</span> <span class="nf">max_partition</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">schema</span><span class="p">,</span> <span class="n">table_name</span><span class="p">,</span> <span class="n">field</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="nb">filter</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns the maximum value for all partitions in a table. Works only</span>
+<span class="sd"> for tables that have a single partition key. For subpartitioned</span>
+<span class="sd"> table, we recommend using signal tables.</span>
+
+<span class="sd"> >>> hh = HiveMetastoreHook()</span>
+<span class="sd"> >>> t = 'static_babynames_partitioned'</span>
+<span class="sd"> >>> hh.max_partition(schema='airflow', table_name=t)</span>
+<span class="sd"> '2015-01-01'</span>
+<span class="sd"> """</span>
+ <span class="n">parts</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_partitions</span><span class="p">(</span><span class="n">schema</span><span class="p">,</span> <span class="n">table_name</span><span class="p">,</span> <span class="nb">filter</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">parts</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">None</span>
+ <span class="k">elif</span> <span class="nb">len</span><span class="p">(</span><span class="n">parts</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
+ <span class="n">field</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">parts</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">.</span><span class="n">keys</span><span class="p">())[</span><span class="mi">0</span><span class="p">]</span>
+ <span class="k">elif</span> <span class="ow">not</span> <span class="n">field</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span>
+ <span class="s2">"Please specify the field you want the max "</span>
+ <span class="s2">"value for"</span><span class="p">)</span>
+
+ <span class="k">return</span> <span class="nb">max</span><span class="p">([</span><span class="n">p</span><span class="p">[</span><span class="n">field</span><span class="p">]</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">parts</span><span class="p">])</span></div>
+
+<div class="viewcode-block" id="HiveMetastoreHook.table_exists"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveMetastoreHook.table_exists">[docs]</a> <span class="k">def</span> <span class="nf">table_exists</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">table_name</span><span class="p">,</span> <span class="n">db</span><span class="o">=</span><span class="s1">'default'</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Check if table exists</span>
+
+<span class="sd"> >>> hh = HiveMetastoreHook()</span>
+<span class="sd"> >>> hh.table_exists(db='airflow', table_name='static_babynames')</span>
+<span class="sd"> True</span>
+<span class="sd"> >>> hh.table_exists(db='airflow', table_name='does_not_exist')</span>
+<span class="sd"> False</span>
+<span class="sd"> """</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">t</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_table</span><span class="p">(</span><span class="n">table_name</span><span class="p">,</span> <span class="n">db</span><span class="p">)</span>
+ <span class="k">return</span> <span class="bp">True</span>
+ <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">False</span></div></div>
+
+
+<div class="viewcode-block" id="HiveServer2Hook"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveServer2Hook">[docs]</a><span class="k">class</span> <span class="nc">HiveServer2Hook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Wrapper around the impyla library</span>
+
+<span class="sd"> Note that the default authMechanism is PLAIN, to override it you</span>
+<span class="sd"> can specify it in the ``extra`` of your connection in the UI as in</span>
+<span class="sd"> """</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hiveserver2_conn_id</span><span class="o">=</span><span class="s1">'hiveserver2_default'</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hiveserver2_conn_id</span> <span class="o">=</span> <span class="n">hiveserver2_conn_id</span>
+
+ <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">db</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">hiveserver2_conn_id</span><span class="p">)</span>
+ <span class="n">auth_mechanism</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'authMechanism'</span><span class="p">,</span> <span class="s1">'PLAIN'</span><span class="p">)</span>
+ <span class="n">kerberos_service_name</span> <span class="o">=</span> <span class="bp">None</span>
+ <span class="k">if</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'security'</span><span class="p">)</span> <span class="o">==</span> <span class="s1">'kerberos'</span><span class="p">:</span>
+ <span class="n">auth_mechanism</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'authMechanism'</span><span class="p">,</span> <span class="s1">'GSSAPI'</span><span class="p">)</span>
+ <span class="n">kerberos_service_name</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">extra_dejson</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'kerberos_service_name'</span><span class="p">,</span> <span class="s1">'hive'</span><span class="p">)</span>
+
+ <span class="c1"># impyla uses GSSAPI instead of KERBEROS as a auth_mechanism identifier</span>
+ <span class="k">if</span> <span class="n">auth_mechanism</span> <span class="o">==</span> <span class="s1">'KERBEROS'</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"Detected deprecated 'KERBEROS' for authMechanism for </span><span class="si">%s</span><span class="s2">. Please use 'GSSAPI' instead"</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hiveserver2_conn_id</span><span class="p">)</span>
+ <span class="n">auth_mechanism</span> <span class="o">=</span> <span class="s1">'GSSAPI'</span>
+
+ <span class="kn">from</span> <span class="nn">impala.dbapi</span> <span class="kn">import</span> <span class="n">connect</span>
+ <span class="k">return</span> <span class="n">connect</span><span class="p">(</span>
+ <span class="n">host</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">host</span><span class="p">,</span>
+ <span class="n">port</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">port</span><span class="p">,</span>
+ <span class="n">auth_mechanism</span><span class="o">=</span><span class="n">auth_mechanism</span><span class="p">,</span>
+ <span class="n">kerberos_service_name</span><span class="o">=</span><span class="n">kerberos_service_name</span><span class="p">,</span>
+ <span class="n">user</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">login</span><span class="p">,</span>
+ <span class="n">database</span><span class="o">=</span><span class="n">db</span><span class="o">.</span><span class="n">schema</span> <span class="ow">or</span> <span class="s1">'default'</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">get_results</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="s1">'default'</span><span class="p">,</span> <span class="n">arraysize</span><span class="o">=</span><span class="mi">1000</span><span class="p">):</span>
+ <span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
+ <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">hql</span><span class="p">,</span> <span class="nb">basestring</span><span class="p">):</span>
+ <span class="n">hql</span> <span class="o">=</span> <span class="p">[</span><span class="n">hql</span><span class="p">]</span>
+ <span class="n">results</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'data'</span><span class="p">:</span> <span class="p">[],</span>
+ <span class="s1">'header'</span><span class="p">:</span> <span class="p">[],</span>
+ <span class="p">}</span>
+ <span class="k">for</span> <span class="n">statement</span> <span class="ow">in</span> <span class="n">hql</span><span class="p">:</span>
+ <span class="k">with</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span> <span class="k">as</span> <span class="n">cur</span><span class="p">:</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">statement</span><span class="p">)</span>
+ <span class="n">records</span> <span class="o">=</span> <span class="n">cur</span><span class="o">.</span><span class="n">fetchall</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">records</span><span class="p">:</span>
+ <span class="n">results</span> <span class="o">=</span> <span class="p">{</span>
+ <span class="s1">'data'</span><span class="p">:</span> <span class="n">records</span><span class="p">,</span>
+ <span class="s1">'header'</span><span class="p">:</span> <span class="n">cur</span><span class="o">.</span><span class="n">description</span><span class="p">,</span>
+ <span class="p">}</span>
+ <span class="k">return</span> <span class="n">results</span>
+
+ <span class="k">def</span> <span class="nf">to_csv</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">hql</span><span class="p">,</span>
+ <span class="n">csv_filepath</span><span class="p">,</span>
+ <span class="n">schema</span><span class="o">=</span><span class="s1">'default'</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="s1">','</span><span class="p">,</span>
+ <span class="n">lineterminator</span><span class="o">=</span><span class="s1">'</span><span class="se">\r\n</span><span class="s1">'</span><span class="p">,</span>
+ <span class="n">output_header</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
+ <span class="n">fetch_size</span><span class="o">=</span><span class="mi">1000</span><span class="p">):</span>
+ <span class="n">schema</span> <span class="o">=</span> <span class="n">schema</span> <span class="ow">or</span> <span class="s1">'default'</span>
+ <span class="k">with</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span> <span class="k">as</span> <span class="n">conn</span><span class="p">:</span>
+ <span class="k">with</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span> <span class="k">as</span> <span class="n">cur</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running query: "</span> <span class="o">+</span> <span class="n">hql</span><span class="p">)</span>
+ <span class="n">cur</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">hql</span><span class="p">)</span>
+ <span class="n">schema</span> <span class="o">=</span> <span class="n">cur</span><span class="o">.</span><span class="n">description</span>
+ <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">csv_filepath</span><span class="p">,</span> <span class="s1">'wb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+ <span class="n">writer</span> <span class="o">=</span> <span class="n">csv</span><span class="o">.</span><span class="n">writer</span><span class="p">(</span><span class="n">f</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">,</span>
+ <span class="n">lineterminator</span><span class="o">=</span><span class="n">lineterminator</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s1">'utf-8'</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">output_header</span><span class="p">:</span>
+ <span class="n">writer</span><span class="o">.</span><span class="n">writerow</span><span class="p">([</span><span class="n">c</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
+ <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">cur</span><span class="o">.</span><span class="n">description</span><span class="p">])</span>
+ <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
+ <span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
+ <span class="n">rows</span> <span class="o">=</span> <span class="p">[</span><span class="n">row</span> <span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">cur</span><span class="o">.</span><span class="n">fetchmany</span><span class="p">(</span><span class="n">fetch_size</span><span class="p">)</span> <span class="k">if</span> <span class="n">row</span><span class="p">]</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">rows</span><span class="p">:</span>
+ <span class="k">break</span>
+
+ <span class="n">writer</span><span class="o">.</span><span class="n">writerows</span><span class="p">(</span><span class="n">rows</span><span class="p">)</span>
+ <span class="n">i</span> <span class="o">+=</span> <span class="nb">len</span><span class="p">(</span><span class="n">rows</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Written {0} rows so far."</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">i</span><span class="p">))</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Done. Loaded a total of {0} rows."</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">i</span><span class="p">))</span>
+
+<div class="viewcode-block" id="HiveServer2Hook.get_records"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveServer2Hook.get_records">[docs]</a> <span class="k">def</span> <span class="nf">get_records</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="s1">'default'</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Get a set of records from a Hive query.</span>
+
+<span class="sd"> >>> hh = HiveServer2Hook()</span>
+<span class="sd"> >>> sql = "SELECT * FROM airflow.static_babynames LIMIT 100"</span>
+<span class="sd"> >>> len(hh.get_records(sql))</span>
+<span class="sd"> 100</span>
+<span class="sd"> """</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_results</span><span class="p">(</span><span class="n">hql</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">)[</span><span class="s1">'data'</span><span class="p">]</span></div>
+
+<div class="viewcode-block" id="HiveServer2Hook.get_pandas_df"><a class="viewcode-back" href="../code.html#airflow.hooks.HiveServer2Hook.get_pandas_df">[docs]</a> <span class="k">def</span> <span class="nf">get_pandas_df</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hql</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="s1">'default'</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Get a pandas dataframe from a Hive query</span>
+
+<span class="sd"> >>> hh = HiveServer2Hook()</span>
+<span class="sd"> >>> sql = "SELECT * FROM airflow.static_babynames LIMIT 100"</span>
+<span class="sd"> >>> df = hh.get_pandas_df(sql)</span>
+<span class="sd"> >>> len(df.index)</span>
+<span class="sd"> 100</span>
+<span class="sd"> """</span>
+ <span class="kn">import</span> <span class="nn">pandas</span> <span class="kn">as</span> <span class="nn">pd</span>
+ <span class="n">res</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_results</span><span class="p">(</span><span class="n">hql</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="n">schema</span><span class="p">)</span>
+ <span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">(</span><span class="n">res</span><span class="p">[</span><span class="s1">'data'</span><span class="p">])</span>
+ <span class="n">df</span><span class="o">.</span><span class="n">columns</span> <span class="o">=</span> <span class="p">[</span><span class="n">c</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">res</span><span class="p">[</span><span class="s1">'header'</span><span class="p">]]</span>
+ <span class="k">return</span> <span class="n">df</span></div></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[26/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/gcs_hook.html
----------------------------------------------------------------------
diff --git a/_modules/gcs_hook.html b/_modules/gcs_hook.html
new file mode 100644
index 0000000..f110dd2
--- /dev/null
+++ b/_modules/gcs_hook.html
@@ -0,0 +1,296 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>gcs_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>gcs_hook</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for gcs_hook</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="c1">#</span>
+
+<span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">from</span> <span class="nn">airflow.contrib.hooks.gc_base_hook</span> <span class="kn">import</span> <span class="n">GoogleCloudBaseHook</span>
+<span class="kn">from</span> <span class="nn">apiclient.discovery</span> <span class="kn">import</span> <span class="n">build</span>
+<span class="kn">from</span> <span class="nn">apiclient.http</span> <span class="kn">import</span> <span class="n">MediaFileUpload</span>
+
+<span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s2">"google_cloud_storage"</span><span class="p">)</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
+
+
+<div class="viewcode-block" id="GoogleCloudStorageHook"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.GoogleCloudStorageHook">[docs]</a><span class="k">class</span> <span class="nc">GoogleCloudStorageHook</span><span class="p">(</span><span class="n">GoogleCloudBaseHook</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Interact with Google Cloud Storage. Connections must be defined with an</span>
+<span class="sd"> extras JSON field containing:</span>
+
+<span class="sd"> ::</span>
+<span class="sd"> {</span>
+<span class="sd"> "project": "<google project ID>",</span>
+<span class="sd"> "service_account": "<google service account email>",</span>
+<span class="sd"> "key_path": "<p12 key path>"</span>
+<span class="sd"> }</span>
+
+<span class="sd"> If you have used ``gcloud auth`` to authenticate on the machine that's</span>
+<span class="sd"> running Airflow, you can exclude the service_account and key_path</span>
+<span class="sd"> parameters.</span>
+<span class="sd"> """</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">google_cloud_storage_conn_id</span><span class="o">=</span><span class="s1">'google_cloud_storage_default'</span><span class="p">,</span>
+ <span class="n">delegate_to</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">GoogleCloudStorageHook</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="n">google_cloud_storage_conn_id</span><span class="p">,</span> <span class="n">delegate_to</span><span class="p">)</span>
+
+<div class="viewcode-block" id="GoogleCloudStorageHook.get_conn"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.GoogleCloudStorageHook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a Google Cloud Storage service object.</span>
+<span class="sd"> """</span>
+ <span class="n">http_authorized</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_authorize</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">build</span><span class="p">(</span><span class="s1">'storage'</span><span class="p">,</span> <span class="s1">'v1'</span><span class="p">,</span> <span class="n">http</span><span class="o">=</span><span class="n">http_authorized</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="GoogleCloudStorageHook.download"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.GoogleCloudStorageHook.download">[docs]</a> <span class="k">def</span> <span class="nf">download</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="p">,</span> <span class="n">filename</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Get a file from Google Cloud Storage.</span>
+
+<span class="sd"> :param bucket: The bucket to fetch from.</span>
+<span class="sd"> :type bucket: string</span>
+<span class="sd"> :param object: The object to fetch.</span>
+<span class="sd"> :type object: string</span>
+<span class="sd"> :param filename: If set, a local file path where the file should be written to.</span>
+<span class="sd"> :type filename: string</span>
+<span class="sd"> """</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">downloaded_file_bytes</span> <span class="o">=</span> <span class="n">service</span> \
+ <span class="o">.</span><span class="n">objects</span><span class="p">()</span> \
+ <span class="o">.</span><span class="n">get_media</span><span class="p">(</span><span class="n">bucket</span><span class="o">=</span><span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="o">=</span><span class="nb">object</span><span class="p">)</span> \
+ <span class="o">.</span><span class="n">execute</span><span class="p">()</span>
+
+ <span class="c1"># Write the file to local file path, if requested.</span>
+ <span class="k">if</span> <span class="n">filename</span><span class="p">:</span>
+ <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">file_fd</span><span class="p">:</span>
+ <span class="n">file_fd</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">downloaded_file_bytes</span><span class="p">)</span>
+
+ <span class="k">return</span> <span class="n">downloaded_file_bytes</span></div>
+
+<div class="viewcode-block" id="GoogleCloudStorageHook.upload"><a class="viewcode-back" href="../code.html#airflow.contrib.hooks.GoogleCloudStorageHook.upload">[docs]</a> <span class="k">def</span> <span class="nf">upload</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket</span><span class="p">,</span> <span class="nb">object</span><span class="p">,</span> <span class="n">filename</span><span class="p">,</span> <span class="n">mime_type</span><span class="o">=</span><span class="s1">'application/octet-stream'</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Uploads a local file to Google Cloud Storage.</span>
+
+<span class="sd"> :param bucket: The bucket to upload to.</span>
+<span class="sd"> :type bucket: string</span>
+<span class="sd"> :param object: The object name to set when uploading the local file.</span>
+<span class="sd"> :type object: string</span>
+<span class="sd"> :param filename: The local file path to the file to be uploaded.</span>
+<span class="sd"> :type filename: string</span>
+<span class="sd"> :param mime_type: The MIME type to set when uploading the file.</span>
+<span class="sd"> :type mime_type: string</span>
+<span class="sd"> """</span>
+ <span class="n">service</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">media</span> <span class="o">=</span> <span class="n">MediaFileUpload</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="n">mime_type</span><span class="p">)</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="n">service</span> \
+ <span class="o">.</span><span class="n">objects</span><span class="p">()</span> \
+ <span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="n">bucket</span><span class="o">=</span><span class="n">bucket</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="nb">object</span><span class="p">,</span> <span class="n">media_body</span><span class="o">=</span><span class="n">media</span><span class="p">)</span> \
+ <span class="o">.</span><span class="n">execute</span><span class="p">()</span></div></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/generic_transfer.html
----------------------------------------------------------------------
diff --git a/_modules/generic_transfer.html b/_modules/generic_transfer.html
new file mode 100644
index 0000000..b4c9d7b
--- /dev/null
+++ b/_modules/generic_transfer.html
@@ -0,0 +1,264 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>generic_transfer — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>generic_transfer</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for generic_transfer</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">logging</span>
+
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="kn">import</span> <span class="n">BaseHook</span>
+
+
+<div class="viewcode-block" id="GenericTransfer"><a class="viewcode-back" href="../code.html#airflow.operators.GenericTransfer">[docs]</a><span class="k">class</span> <span class="nc">GenericTransfer</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Moves data from a connection to another, assuming that they both</span>
+<span class="sd"> provide the required methods in their respective hooks. The source hook</span>
+<span class="sd"> needs to expose a `get_records` method, and the destination a</span>
+<span class="sd"> `insert_rows` method.</span>
+
+<span class="sd"> This is mean to be used on small-ish datasets that fit in memory.</span>
+
+<span class="sd"> :param sql: SQL query to execute against the source database</span>
+<span class="sd"> :type sql: str</span>
+<span class="sd"> :param destination_table: target table</span>
+<span class="sd"> :type destination_table: str</span>
+<span class="sd"> :param source_conn_id: source connection</span>
+<span class="sd"> :type source_conn_id: str</span>
+<span class="sd"> :param destination_conn_id: source connection</span>
+<span class="sd"> :type destination_conn_id: str</span>
+<span class="sd"> :param preoperator: sql statement or list of statements to be</span>
+<span class="sd"> executed prior to loading the data</span>
+<span class="sd"> :type preoperator: str or list of str</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,</span> <span class="s1">'destination_table'</span><span class="p">,</span> <span class="s1">'preoperator'</span><span class="p">)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sql'</span><span class="p">,</span> <span class="s1">'.hql'</span><span class="p">,)</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#b0f07c'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">sql</span><span class="p">,</span>
+ <span class="n">destination_table</span><span class="p">,</span>
+ <span class="n">source_conn_id</span><span class="p">,</span>
+ <span class="n">destination_conn_id</span><span class="p">,</span>
+ <span class="n">preoperator</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">GenericTransfer</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">destination_table</span> <span class="o">=</span> <span class="n">destination_table</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">source_conn_id</span> <span class="o">=</span> <span class="n">source_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">destination_conn_id</span> <span class="o">=</span> <span class="n">destination_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">preoperator</span> <span class="o">=</span> <span class="n">preoperator</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">source_hook</span> <span class="o">=</span> <span class="n">BaseHook</span><span class="o">.</span><span class="n">get_hook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">source_conn_id</span><span class="p">)</span>
+
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Extracting data from {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">source_conn_id</span><span class="p">))</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Executing: </span><span class="se">\n</span><span class="s2">"</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+ <span class="n">results</span> <span class="o">=</span> <span class="n">source_hook</span><span class="o">.</span><span class="n">get_records</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+
+ <span class="n">destination_hook</span> <span class="o">=</span> <span class="n">BaseHook</span><span class="o">.</span><span class="n">get_hook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">destination_conn_id</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">preoperator</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Running preoperator"</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">preoperator</span><span class="p">)</span>
+ <span class="n">destination_hook</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">preoperator</span><span class="p">)</span>
+
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Inserting rows into {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">destination_conn_id</span><span class="p">))</span>
+ <span class="n">destination_hook</span><span class="o">.</span><span class="n">insert_rows</span><span class="p">(</span><span class="n">table</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">destination_table</span><span class="p">,</span> <span class="n">rows</span><span class="o">=</span><span class="n">results</span><span class="p">)</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[15/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/css/theme.css
----------------------------------------------------------------------
diff --git a/_static/css/theme.css b/_static/css/theme.css
new file mode 100644
index 0000000..7be9339
--- /dev/null
+++ b/_static/css/theme.css
@@ -0,0 +1,5 @@
+\ufeff*{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}article,aside,details,figcaption,figure,footer,header,hgroup,nav,section{display:block}audio,canvas,video{display:inline-block;*display:inline;*zoom:1}audio:not([controls]){display:none}[hidden]{display:none}*{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}html{font-size:100%;-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%}body{margin:0}a:hover,a:active{outline:0}abbr[title]{border-bottom:1px dotted}b,strong{font-weight:bold}blockquote{margin:0}dfn{font-style:italic}ins{background:#ff9;color:#000;text-decoration:none}mark{background:#ff0;color:#000;font-style:italic;font-weight:bold}pre,code,.rst-content tt,.rst-content code,kbd,samp{font-family:monospace,serif;_font-family:"courier new",monospace;font-size:1em}pre{white-space:pre}q{quotes:none}q:before,q:after{content:"";content:none}small{font-size:85%}sub,sup{font-size:75%;line-height:0;position:relative;ver
tical-align:baseline}sup{top:-0.5em}sub{bottom:-0.25em}ul,ol,dl{margin:0;padding:0;list-style:none;list-style-image:none}li{list-style:none}dd{margin:0}img{border:0;-ms-interpolation-mode:bicubic;vertical-align:middle;max-width:100%}svg:not(:root){overflow:hidden}figure{margin:0}form{margin:0}fieldset{border:0;margin:0;padding:0}label{cursor:pointer}legend{border:0;*margin-left:-7px;padding:0;white-space:normal}button,input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}button,input{line-height:normal}button,input[type="button"],input[type="reset"],input[type="submit"]{cursor:pointer;-webkit-appearance:button;*overflow:visible}button[disabled],input[disabled]{cursor:default}input[type="checkbox"],input[type="radio"]{box-sizing:border-box;padding:0;*width:13px;*height:13px}input[type="search"]{-webkit-appearance:textfield;-moz-box-sizing:content-box;-webkit-box-sizing:content-box;box-sizing:content-box}input[type="search"]::-webkit-search-decor
ation,input[type="search"]::-webkit-search-cancel-button{-webkit-appearance:none}button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}textarea{overflow:auto;vertical-align:top;resize:vertical}table{border-collapse:collapse;border-spacing:0}td{vertical-align:top}.chromeframe{margin:0.2em 0;background:#ccc;color:#000;padding:0.2em 0}.ir{display:block;border:0;text-indent:-999em;overflow:hidden;background-color:transparent;background-repeat:no-repeat;text-align:left;direction:ltr;*line-height:0}.ir br{display:none}.hidden{display:none !important;visibility:hidden}.visuallyhidden{border:0;clip:rect(0 0 0 0);height:1px;margin:-1px;overflow:hidden;padding:0;position:absolute;width:1px}.visuallyhidden.focusable:active,.visuallyhidden.focusable:focus{clip:auto;height:auto;margin:0;overflow:visible;position:static;width:auto}.invisible{visibility:hidden}.relative{position:relative}big,small{font-size:100%}@media print{html,body,section{background:none !important}*{box-shadow:n
one !important;text-shadow:none !important;filter:none !important;-ms-filter:none !important}a,a:visited{text-decoration:underline}.ir a:after,a[href^="javascript:"]:after,a[href^="#"]:after{content:""}pre,blockquote{page-break-inside:avoid}thead{display:table-header-group}tr,img{page-break-inside:avoid}img{max-width:100% !important}@page{margin:0.5cm}p,h2,.rst-content .toctree-wrapper p.caption,h3{orphans:3;widows:3}h2,.rst-content .toctree-wrapper p.caption,h3{page-break-after:avoid}}.fa:before,.wy-menu-vertical li span.toctree-expand:before,.wy-menu-vertical li.on a span.toctree-expand:before,.wy-menu-vertical li.current>a span.toctree-expand:before,.rst-content .admonition-title:before,.rst-content h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3 .headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5 .headerlink:before,.rst-content h6 .headerlink:before,.rst-content dl dt .headerlink:before,.rst-content p.caption .headerlink:before,.rst-con
tent tt.download span:first-child:before,.rst-content code.download span:first-child:before,.icon:before,.wy-dropdown .caret:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-alert,.rst-content .note,.rst-content .attention,.rst-content .caution,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .tip,.rst-content .warning,.rst-content .seealso,.rst-content .admonition-todo,.btn,input[type="text"],input[type="password"],input[type="email"],input[type="url"],input[type="date"],input[type="month"],input[type="time"],input[type="datetime"],input[type="datetime-local"],input[type="week"],input[type="number"],input[type="search"],input[type="tel"],input[type="color"],select,textarea,.wy-menu-vertical li
.on a,.wy-menu-vertical li.current>a,.wy-side-nav-search>a,.wy-side-nav-search .wy-dropdown>a,.wy-nav-top a{-webkit-font-smoothing:antialiased}.clearfix{*zoom:1}.clearfix:before,.clearfix:after{display:table;content:""}.clearfix:after{clear:both}/*!
+ * Font Awesome 4.2.0 by @davegandy - http://fontawesome.io - @fontawesome
+ * License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License)
+ */@font-face{font-family:'FontAwesome';src:url("../fonts/fontawesome-webfont.eot?v=4.2.0");src:url("../fonts/fontawesome-webfont.eot?#iefix&v=4.2.0") format("embedded-opentype"),url("../fonts/fontawesome-webfont.woff?v=4.2.0") format("woff"),url("../fonts/fontawesome-webfont.ttf?v=4.2.0") format("truetype"),url("../fonts/fontawesome-webfont.svg?v=4.2.0#fontawesomeregular") format("svg");font-weight:normal;font-style:normal}.fa,.wy-menu-vertical li span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li.current>a span.toctree-expand,.rst-content .admonition-title,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content dl dt .headerlink,.rst-content p.caption .headerlink,.rst-content tt.download span:first-child,.rst-content code.download span:first-child,.icon{display:inline-block;font:normal normal normal 14px/1 FontAwesome;font
-size:inherit;text-rendering:auto;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}.fa-lg{font-size:1.33333em;line-height:0.75em;vertical-align:-15%}.fa-2x{font-size:2em}.fa-3x{font-size:3em}.fa-4x{font-size:4em}.fa-5x{font-size:5em}.fa-fw{width:1.28571em;text-align:center}.fa-ul{padding-left:0;margin-left:2.14286em;list-style-type:none}.fa-ul>li{position:relative}.fa-li{position:absolute;left:-2.14286em;width:2.14286em;top:0.14286em;text-align:center}.fa-li.fa-lg{left:-1.85714em}.fa-border{padding:.2em .25em .15em;border:solid 0.08em #eee;border-radius:.1em}.pull-right{float:right}.pull-left{float:left}.fa.pull-left,.wy-menu-vertical li span.pull-left.toctree-expand,.wy-menu-vertical li.on a span.pull-left.toctree-expand,.wy-menu-vertical li.current>a span.pull-left.toctree-expand,.rst-content .pull-left.admonition-title,.rst-content h1 .pull-left.headerlink,.rst-content h2 .pull-left.headerlink,.rst-content h3 .pull-left.headerlink,.rst-content h4 .pull-left.he
aderlink,.rst-content h5 .pull-left.headerlink,.rst-content h6 .pull-left.headerlink,.rst-content dl dt .pull-left.headerlink,.rst-content p.caption .pull-left.headerlink,.rst-content tt.download span.pull-left:first-child,.rst-content code.download span.pull-left:first-child,.pull-left.icon{margin-right:.3em}.fa.pull-right,.wy-menu-vertical li span.pull-right.toctree-expand,.wy-menu-vertical li.on a span.pull-right.toctree-expand,.wy-menu-vertical li.current>a span.pull-right.toctree-expand,.rst-content .pull-right.admonition-title,.rst-content h1 .pull-right.headerlink,.rst-content h2 .pull-right.headerlink,.rst-content h3 .pull-right.headerlink,.rst-content h4 .pull-right.headerlink,.rst-content h5 .pull-right.headerlink,.rst-content h6 .pull-right.headerlink,.rst-content dl dt .pull-right.headerlink,.rst-content p.caption .pull-right.headerlink,.rst-content tt.download span.pull-right:first-child,.rst-content code.download span.pull-right:first-child,.pull-right.icon{margin-left
:.3em}.fa-spin{-webkit-animation:fa-spin 2s infinite linear;animation:fa-spin 2s infinite linear}@-webkit-keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}100%{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}@keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}100%{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}.fa-rotate-90{filter:progid:DXImageTransform.Microsoft.BasicImage(rotation=1);-webkit-transform:rotate(90deg);-ms-transform:rotate(90deg);transform:rotate(90deg)}.fa-rotate-180{filter:progid:DXImageTransform.Microsoft.BasicImage(rotation=2);-webkit-transform:rotate(180deg);-ms-transform:rotate(180deg);transform:rotate(180deg)}.fa-rotate-270{filter:progid:DXImageTransform.Microsoft.BasicImage(rotation=3);-webkit-transform:rotate(270deg);-ms-transform:rotate(270deg);transform:rotate(270deg)}.fa-flip-horizontal{filter:progid:DXImageTransform.Microsoft.BasicImage(rotation=0);-webkit-transform:scale(-1, 1);-ms
-transform:scale(-1, 1);transform:scale(-1, 1)}.fa-flip-vertical{filter:progid:DXImageTransform.Microsoft.BasicImage(rotation=2);-webkit-transform:scale(1, -1);-ms-transform:scale(1, -1);transform:scale(1, -1)}:root .fa-rotate-90,:root .fa-rotate-180,:root .fa-rotate-270,:root .fa-flip-horizontal,:root .fa-flip-vertical{filter:none}.fa-stack{position:relative;display:inline-block;width:2em;height:2em;line-height:2em;vertical-align:middle}.fa-stack-1x,.fa-stack-2x{position:absolute;left:0;width:100%;text-align:center}.fa-stack-1x{line-height:inherit}.fa-stack-2x{font-size:2em}.fa-inverse{color:#fff}.fa-glass:before{content:"\uf000"}.fa-music:before{content:"\uf001"}.fa-search:before,.icon-search:before{content:"\uf002"}.fa-envelope-o:before{content:"\uf003"}.fa-heart:before{content:"\uf004"}.fa-star:before{content:"\uf005"}.fa-star-o:before{content:"\uf006"}.fa-user:before{content:"\uf007"}.fa-film:before{content:"\uf008"}.fa-th-large:before{content:"\uf009"}.fa-th:before{content:"\uf00a"}.fa-th-list:before{content:
"\uf00b"}.fa-check:before{content:"\uf00c"}.fa-remove:before,.fa-close:before,.fa-times:before{content:"\uf00d"}.fa-search-plus:before{content:"\uf00e"}.fa-search-minus:before{content:"\uf010"}.fa-power-off:before{content:"\uf011"}.fa-signal:before{content:"\uf012"}.fa-gear:before,.fa-cog:before{content:"\uf013"}.fa-trash-o:before{content:"\uf014"}.fa-home:before,.icon-home:before{content:"\uf015"}.fa-file-o:before{content:"\uf016"}.fa-clock-o:before{content:"\uf017"}.fa-road:before{content:"\uf018"}.fa-download:before,.rst-content tt.download span:first-child:before,.rst-content code.download span:first-child:before{content:"\uf019"}.fa-arrow-circle-o-down:before{content:"\uf01a"}.fa-arrow-circle-o-up:before{content:"\uf01b"}.fa-inbox:before{content:"\uf01c"}.fa-play-circle-o:before{content:"\uf01d"}.fa-rotate-right:before,.fa-repeat:before{content:"\uf01e"}.fa-refresh:before{content:"\uf021"}.fa-list-alt:before{content:"\uf022"}.fa-lock:before{content:"\uf023"}.fa-flag:before{content:"\uf024"}.fa-headphones:before{content:"\uf025"}.fa-volume-off:befor
e{content:"\uf026"}.fa-volume-down:before{content:"\uf027"}.fa-volume-up:before{content:"\uf028"}.fa-qrcode:before{content:"\uf029"}.fa-barcode:before{content:"\uf02a"}.fa-tag:before{content:"\uf02b"}.fa-tags:before{content:"\uf02c"}.fa-book:before,.icon-book:before{content:"\uf02d"}.fa-bookmark:before{content:"\uf02e"}.fa-print:before{content:"\uf02f"}.fa-camera:before{content:"\uf030"}.fa-font:before{content:"\uf031"}.fa-bold:before{content:"\uf032"}.fa-italic:before{content:"\uf033"}.fa-text-height:before{content:"\uf034"}.fa-text-width:before{content:"\uf035"}.fa-align-left:before{content:"\uf036"}.fa-align-center:before{content:"\uf037"}.fa-align-right:before{content:"\uf038"}.fa-align-justify:before{content:"\uf039"}.fa-list:before{content:"\uf03a"}.fa-dedent:before,.fa-outdent:before{content:"\uf03b"}.fa-indent:before{content:"\uf03c"}.fa-video-camera:before{content:"\uf03d"}.fa-photo:before,.fa-image:before,.fa-picture-o:before{content:"\uf03e"}.fa-pencil:before{content:"\uf040"}.fa-map-marker:before{content:"\uf041"}.fa-adjust:before{content:"\uf042"}.fa-t
int:before{content:"\uf043"}.fa-edit:before,.fa-pencil-square-o:before{content:"\uf044"}.fa-share-square-o:before{content:"\uf045"}.fa-check-square-o:before{content:"\uf046"}.fa-arrows:before{content:"\uf047"}.fa-step-backward:before{content:"\uf048"}.fa-fast-backward:before{content:"\uf049"}.fa-backward:before{content:"\uf04a"}.fa-play:before{content:"\uf04b"}.fa-pause:before{content:"\uf04c"}.fa-stop:before{content:"\uf04d"}.fa-forward:before{content:"\uf04e"}.fa-fast-forward:before{content:"\uf050"}.fa-step-forward:before{content:"\uf051"}.fa-eject:before{content:"\uf052"}.fa-chevron-left:before{content:"\uf053"}.fa-chevron-right:before{content:"\uf054"}.fa-plus-circle:before{content:"\uf055"}.fa-minus-circle:before{content:"\uf056"}.fa-times-circle:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before{content:"\uf057"}.fa-check-circle:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before{content:"\uf058"}.fa-question-circle:before{content:"\uf059"}.fa-info-circle:before{content:"\uf05a"}.fa-crossha
irs:before{content:"\uf05b"}.fa-times-circle-o:before{content:"\uf05c"}.fa-check-circle-o:before{content:"\uf05d"}.fa-ban:before{content:"\uf05e"}.fa-arrow-left:before{content:"\uf060"}.fa-arrow-right:before{content:"\uf061"}.fa-arrow-up:before{content:"\uf062"}.fa-arrow-down:before{content:"\uf063"}.fa-mail-forward:before,.fa-share:before{content:"\uf064"}.fa-expand:before{content:"\uf065"}.fa-compress:before{content:"\uf066"}.fa-plus:before{content:"\uf067"}.fa-minus:before{content:"\uf068"}.fa-asterisk:before{content:"\uf069"}.fa-exclamation-circle:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.rst-content .admonition-title:before{content:"\uf06a"}.fa-gift:before{content:"\uf06b"}.fa-leaf:before{content:"\uf06c"}.fa-fire:before,.icon-fire:before{content:"\uf06d"}.fa-eye:before{content:"\uf06e"}.fa-eye-slash:before{content:"\uf070"}.fa-warning:before,.fa-exclamation-triangle:before{content:"\uf071"}.fa-plane:before{content:"\uf072"}.fa-calendar:
before{content:"\uf073"}.fa-random:before{content:"\uf074"}.fa-comment:before{content:"\uf075"}.fa-magnet:before{content:"\uf076"}.fa-chevron-up:before{content:"\uf077"}.fa-chevron-down:before{content:"\uf078"}.fa-retweet:before{content:"\uf079"}.fa-shopping-cart:before{content:"\uf07a"}.fa-folder:before{content:"\uf07b"}.fa-folder-open:before{content:"\uf07c"}.fa-arrows-v:before{content:"\uf07d"}.fa-arrows-h:before{content:"\uf07e"}.fa-bar-chart-o:before,.fa-bar-chart:before{content:"\uf080"}.fa-twitter-square:before{content:"\uf081"}.fa-facebook-square:before{content:"\uf082"}.fa-camera-retro:before{content:"\uf083"}.fa-key:before{content:"\uf084"}.fa-gears:before,.fa-cogs:before{content:"\uf085"}.fa-comments:before{content:"\uf086"}.fa-thumbs-o-up:before{content:"\uf087"}.fa-thumbs-o-down:before{content:"\uf088"}.fa-star-half:before{content:"\uf089"}.fa-heart-o:before{content:"\uf08a"}.fa-sign-out:before{content:"\uf08b"}.fa-linkedin-square:before{content:"\uf08c"}.fa-thumb-tack:before{content:"\uf08d"}.fa-external-link:before{content:"\uf08e"}.fa-sign-in:bef
ore{content:"\uf090"}.fa-trophy:before{content:"\uf091"}.fa-github-square:before{content:"\uf092"}.fa-upload:before{content:"\uf093"}.fa-lemon-o:before{content:"\uf094"}.fa-phone:before{content:"\uf095"}.fa-square-o:before{content:"\uf096"}.fa-bookmark-o:before{content:"\uf097"}.fa-phone-square:before{content:"\uf098"}.fa-twitter:before{content:"\uf099"}.fa-facebook:before{content:"\uf09a"}.fa-github:before,.icon-github:before{content:"\uf09b"}.fa-unlock:before{content:"\uf09c"}.fa-credit-card:before{content:"\uf09d"}.fa-rss:before{content:"\uf09e"}.fa-hdd-o:before{content:"\uf0a0"}.fa-bullhorn:before{content:"\uf0a1"}.fa-bell:before{content:"\uf0f3"}.fa-certificate:before{content:"\uf0a3"}.fa-hand-o-right:before{content:"\uf0a4"}.fa-hand-o-left:before{content:"\uf0a5"}.fa-hand-o-up:before{content:"\uf0a6"}.fa-hand-o-down:before{content:"\uf0a7"}.fa-arrow-circle-left:before,.icon-circle-arrow-left:before{content:"\uf0a8"}.fa-arrow-circle-right:before,.icon-circle-arrow-right:before{content:"\uf0a9"}.fa-arrow-circle-up:before{content:"\uf0aa"}.fa-arrow-circle-
down:before{content:"\uf0ab"}.fa-globe:before{content:"\uf0ac"}.fa-wrench:before{content:"\uf0ad"}.fa-tasks:before{content:"\uf0ae"}.fa-filter:before{content:"\uf0b0"}.fa-briefcase:before{content:"\uf0b1"}.fa-arrows-alt:before{content:"\uf0b2"}.fa-group:before,.fa-users:before{content:"\uf0c0"}.fa-chain:before,.fa-link:before,.icon-link:before{content:"\uf0c1"}.fa-cloud:before{content:"\uf0c2"}.fa-flask:before{content:"\uf0c3"}.fa-cut:before,.fa-scissors:before{content:"\uf0c4"}.fa-copy:before,.fa-files-o:before{content:"\uf0c5"}.fa-paperclip:before{content:"\uf0c6"}.fa-save:before,.fa-floppy-o:before{content:"\uf0c7"}.fa-square:before{content:"\uf0c8"}.fa-navicon:before,.fa-reorder:before,.fa-bars:before{content:"\uf0c9"}.fa-list-ul:before{content:"\uf0ca"}.fa-list-ol:before{content:"\uf0cb"}.fa-strikethrough:before{content:"\uf0cc"}.fa-underline:before{content:"\uf0cd"}.fa-table:before{content:"\uf0ce"}.fa-magic:before{content:"\uf0d0"}.fa-truck:before{content:"\uf0d1"}.fa-pinterest:before{content:"\uf0d2"}.fa-pinterest-square:before{content:"\uf0d3"}.fa-go
ogle-plus-square:before{content:"\uf0d4"}.fa-google-plus:before{content:"\uf0d5"}.fa-money:before{content:"\uf0d6"}.fa-caret-down:before,.wy-dropdown .caret:before,.icon-caret-down:before{content:"\uf0d7"}.fa-caret-up:before{content:"\uf0d8"}.fa-caret-left:before{content:"\uf0d9"}.fa-caret-right:before{content:"\uf0da"}.fa-columns:before{content:"\uf0db"}.fa-unsorted:before,.fa-sort:before{content:"\uf0dc"}.fa-sort-down:before,.fa-sort-desc:before{content:"\uf0dd"}.fa-sort-up:before,.fa-sort-asc:before{content:"\uf0de"}.fa-envelope:before{content:"\uf0e0"}.fa-linkedin:before{content:"\uf0e1"}.fa-rotate-left:before,.fa-undo:before{content:"\uf0e2"}.fa-legal:before,.fa-gavel:before{content:"\uf0e3"}.fa-dashboard:before,.fa-tachometer:before{content:"\uf0e4"}.fa-comment-o:before{content:"\uf0e5"}.fa-comments-o:before{content:"\uf0e6"}.fa-flash:before,.fa-bolt:before{content:"\uf0e7"}.fa-sitemap:before{content:"\uf0e8"}.fa-umbrella:before{content:"\uf0e9"}.fa-paste:before,.fa-clipboard:before{content:"\uf0ea"}.fa-lightbulb-o:before{content:"\uf0eb"}.fa-e
xchange:before{content:"\uf0ec"}.fa-cloud-download:before{content:"\uf0ed"}.fa-cloud-upload:before{content:"\uf0ee"}.fa-user-md:before{content:"\uf0f0"}.fa-stethoscope:before{content:"\uf0f1"}.fa-suitcase:before{content:"\uf0f2"}.fa-bell-o:before{content:"\uf0a2"}.fa-coffee:before{content:"\uf0f4"}.fa-cutlery:before{content:"\uf0f5"}.fa-file-text-o:before{content:"\uf0f6"}.fa-building-o:before{content:"\uf0f7"}.fa-hospital-o:before{content:"\uf0f8"}.fa-ambulance:before{content:"\uf0f9"}.fa-medkit:before{content:"\uf0fa"}.fa-fighter-jet:before{content:"\uf0fb"}.fa-beer:before{content:"\uf0fc"}.fa-h-square:before{content:"\uf0fd"}.fa-plus-square:before{content:"\uf0fe"}.fa-angle-double-left:before{content:"\uf100"}.fa-angle-double-right:before{content:"\uf101"}.fa-angle-double-up:before{content:"\uf102"}.fa-angle-double-down:before{content:"\uf103"}.fa-angle-left:before{content:"\uf104"}.fa-angle-right:before{content:"\uf105"}.fa-angle-up:before{content:"\uf106"}.fa-angle-down:before{content:"\uf107"}.fa-desktop:before{content:"\uf108"}.fa-laptop:before{content:"
"}.fa-tablet:before{content:"\uf10a"}.fa-mobile-phone:before,.fa-mobile:before{content:"\uf10b"}.fa-circle-o:before{content:"\uf10c"}.fa-quote-left:before{content:"\uf10d"}.fa-quote-right:before{content:"\uf10e"}.fa-spinner:before{content:"\uf110"}.fa-circle:before{content:"\uf111"}.fa-mail-reply:before,.fa-reply:before{content:"\uf112"}.fa-github-alt:before{content:"\uf113"}.fa-folder-o:before{content:"\uf114"}.fa-folder-open-o:before{content:"\uf115"}.fa-smile-o:before{content:"\uf118"}.fa-frown-o:before{content:"\uf119"}.fa-meh-o:before{content:"\uf11a"}.fa-gamepad:before{content:"\uf11b"}.fa-keyboard-o:before{content:"\uf11c"}.fa-flag-o:before{content:"\uf11d"}.fa-flag-checkered:before{content:"\uf11e"}.fa-terminal:before{content:"\uf120"}.fa-code:before{content:"\uf121"}.fa-mail-reply-all:before,.fa-reply-all:before{content:"\uf122"}.fa-star-half-empty:before,.fa-star-half-full:before,.fa-star-half-o:before{content:"\uf123"}.fa-location-arrow:before{content:"\uf124"}.fa-crop:before{content:"\uf125"}.fa-code-fork:before{content:"\uf126"}.fa-unlink:be
fore,.fa-chain-broken:before{content:"\uf127"}.fa-question:before{content:"\uf128"}.fa-info:before{content:"\uf129"}.fa-exclamation:before{content:"\uf12a"}.fa-superscript:before{content:"\uf12b"}.fa-subscript:before{content:"\uf12c"}.fa-eraser:before{content:"\uf12d"}.fa-puzzle-piece:before{content:"\uf12e"}.fa-microphone:before{content:"\uf130"}.fa-microphone-slash:before{content:"\uf131"}.fa-shield:before{content:"\uf132"}.fa-calendar-o:before{content:"\uf133"}.fa-fire-extinguisher:before{content:"\uf134"}.fa-rocket:before{content:"\uf135"}.fa-maxcdn:before{content:"\uf136"}.fa-chevron-circle-left:before{content:"\uf137"}.fa-chevron-circle-right:before{content:"\uf138"}.fa-chevron-circle-up:before{content:"\uf139"}.fa-chevron-circle-down:before{content:"\uf13a"}.fa-html5:before{content:"\uf13b"}.fa-css3:before{content:"\uf13c"}.fa-anchor:before{content:"\uf13d"}.fa-unlock-alt:before{content:"\uf13e"}.fa-bullseye:before{content:"\uf140"}.fa-ellipsis-h:before{content:"\uf141"}.fa-ellipsis-v:before{content:"\uf142"}.fa-rss-square:before{content:"\uf143"}.fa-play
-circle:before{content:"\uf144"}.fa-ticket:before{content:"\uf145"}.fa-minus-square:before{content:"\uf146"}.fa-minus-square-o:before,.wy-menu-vertical li.on a span.toctree-expand:before,.wy-menu-vertical li.current>a span.toctree-expand:before{content:"\uf147"}.fa-level-up:before{content:"\uf148"}.fa-level-down:before{content:"\uf149"}.fa-check-square:before{content:"\uf14a"}.fa-pencil-square:before{content:"\uf14b"}.fa-external-link-square:before{content:"\uf14c"}.fa-share-square:before{content:"\uf14d"}.fa-compass:before{content:"\uf14e"}.fa-toggle-down:before,.fa-caret-square-o-down:before{content:"\uf150"}.fa-toggle-up:before,.fa-caret-square-o-up:before{content:"\uf151"}.fa-toggle-right:before,.fa-caret-square-o-right:before{content:"\uf152"}.fa-euro:before,.fa-eur:before{content:"\uf153"}.fa-gbp:before{content:"\uf154"}.fa-dollar:before,.fa-usd:before{content:"\uf155"}.fa-rupee:before,.fa-inr:before{content:"\uf156"}.fa-cny:before,.fa-rmb:before,.fa-yen:before,.fa-jpy:before{content:"\uf157"}.fa-ruble:before,.fa-rouble:before,.fa-r
ub:before{content:"\uf158"}.fa-won:before,.fa-krw:before{content:"\uf159"}.fa-bitcoin:before,.fa-btc:before{content:"\uf15a"}.fa-file:before{content:"\uf15b"}.fa-file-text:before{content:"\uf15c"}.fa-sort-alpha-asc:before{content:"\uf15d"}.fa-sort-alpha-desc:before{content:"\uf15e"}.fa-sort-amount-asc:before{content:"\uf160"}.fa-sort-amount-desc:before{content:"\uf161"}.fa-sort-numeric-asc:before{content:"\uf162"}.fa-sort-numeric-desc:before{content:"\uf163"}.fa-thumbs-up:before{content:"\uf164"}.fa-thumbs-down:before{content:"\uf165"}.fa-youtube-square:before{content:"\uf166"}.fa-youtube:before{content:"\uf167"}.fa-xing:before{content:"\uf168"}.fa-xing-square:before{content:"\uf169"}.fa-youtube-play:before{content:"\uf16a"}.fa-dropbox:before{content:"\uf16b"}.fa-stack-overflow:before{content:"\uf16c"}.fa-instagram:before{content:"\uf16d"}.fa-flickr:before{content:"\uf16e"}.fa-adn:before{content:"\uf170"}.fa-bitbucket:before,.icon-bitbucket:before{content:"\uf171"}.fa-bitbucket-square:before{content:"\uf172"}.fa-tumblr:before{content:"\uf173"}.fa-tumblr-squar
e:before{content:"\uf174"}.fa-long-arrow-down:before{content:"\uf175"}.fa-long-arrow-up:before{content:"\uf176"}.fa-long-arrow-left:before{content:"\uf177"}.fa-long-arrow-right:before{content:"\uf178"}.fa-apple:before{content:"\uf179"}.fa-windows:before{content:"\uf17a"}.fa-android:before{content:"\uf17b"}.fa-linux:before{content:"\uf17c"}.fa-dribbble:before{content:"\uf17d"}.fa-skype:before{content:"\uf17e"}.fa-foursquare:before{content:"\uf180"}.fa-trello:before{content:"\uf181"}.fa-female:before{content:"\uf182"}.fa-male:before{content:"\uf183"}.fa-gittip:before{content:"\uf184"}.fa-sun-o:before{content:"\uf185"}.fa-moon-o:before{content:"\uf186"}.fa-archive:before{content:"\uf187"}.fa-bug:before{content:"\uf188"}.fa-vk:before{content:"\uf189"}.fa-weibo:before{content:"\uf18a"}.fa-renren:before{content:"\uf18b"}.fa-pagelines:before{content:"\uf18c"}.fa-stack-exchange:before{content:"\uf18d"}.fa-arrow-circle-o-right:before{content:"\uf18e"}.fa-arrow-circle-o-left:before{content:"\uf190"}.fa-toggle-left:before,.fa-caret-square-o-left:before{content:"\uf191"}.fa-do
t-circle-o:before{content:"\uf192"}.fa-wheelchair:before{content:"\uf193"}.fa-vimeo-square:before{content:"\uf194"}.fa-turkish-lira:before,.fa-try:before{content:"\uf195"}.fa-plus-square-o:before,.wy-menu-vertical li span.toctree-expand:before{content:"\uf196"}.fa-space-shuttle:before{content:"\uf197"}.fa-slack:before{content:"\uf198"}.fa-envelope-square:before{content:"\uf199"}.fa-wordpress:before{content:"\uf19a"}.fa-openid:before{content:"\uf19b"}.fa-institution:before,.fa-bank:before,.fa-university:before{content:"\uf19c"}.fa-mortar-board:before,.fa-graduation-cap:before{content:"\uf19d"}.fa-yahoo:before{content:"\uf19e"}.fa-google:before{content:"\uf1a0"}.fa-reddit:before{content:"\uf1a1"}.fa-reddit-square:before{content:"\uf1a2"}.fa-stumbleupon-circle:before{content:"\uf1a3"}.fa-stumbleupon:before{content:"\uf1a4"}.fa-delicious:before{content:"\uf1a5"}.fa-digg:before{content:"\uf1a6"}.fa-pied-piper:before{content:"\uf1a7"}.fa-pied-piper-alt:before{content:"\uf1a8"}.fa-drupal:before{content:"\uf1a9"}.fa-joomla:before{content:"\uf1aa"}.fa-language:b
efore{content:"\uf1ab"}.fa-fax:before{content:"\uf1ac"}.fa-building:before{content:"\uf1ad"}.fa-child:before{content:"\uf1ae"}.fa-paw:before{content:"\uf1b0"}.fa-spoon:before{content:"\uf1b1"}.fa-cube:before{content:"\uf1b2"}.fa-cubes:before{content:"\uf1b3"}.fa-behance:before{content:"\uf1b4"}.fa-behance-square:before{content:"\uf1b5"}.fa-steam:before{content:"\uf1b6"}.fa-steam-square:before{content:"\uf1b7"}.fa-recycle:before{content:"\uf1b8"}.fa-automobile:before,.fa-car:before{content:"\uf1b9"}.fa-cab:before,.fa-taxi:before{content:"\uf1ba"}.fa-tree:before{content:"\uf1bb"}.fa-spotify:before{content:"\uf1bc"}.fa-deviantart:before{content:"\uf1bd"}.fa-soundcloud:before{content:"\uf1be"}.fa-database:before{content:"\uf1c0"}.fa-file-pdf-o:before{content:"\uf1c1"}.fa-file-word-o:before{content:"\uf1c2"}.fa-file-excel-o:before{content:"\uf1c3"}.fa-file-powerpoint-o:before{content:"\uf1c4"}.fa-file-photo-o:before,.fa-file-picture-o:before,.fa-file-image-o:before{content:"\uf1c5"}.fa-file-zip-o:before,.fa-file-archive-o:before{content:"\uf1c6"}.fa-file-sound-o:b
efore,.fa-file-audio-o:before{content:"\uf1c7"}.fa-file-movie-o:before,.fa-file-video-o:before{content:"\uf1c8"}.fa-file-code-o:before{content:"\uf1c9"}.fa-vine:before{content:"\uf1ca"}.fa-codepen:before{content:"\uf1cb"}.fa-jsfiddle:before{content:"\uf1cc"}.fa-life-bouy:before,.fa-life-buoy:before,.fa-life-saver:before,.fa-support:before,.fa-life-ring:before{content:"\uf1cd"}.fa-circle-o-notch:before{content:"\uf1ce"}.fa-ra:before,.fa-rebel:before{content:"\uf1d0"}.fa-ge:before,.fa-empire:before{content:"\uf1d1"}.fa-git-square:before{content:"\uf1d2"}.fa-git:before{content:"\uf1d3"}.fa-hacker-news:before{content:"\uf1d4"}.fa-tencent-weibo:before{content:"\uf1d5"}.fa-qq:before{content:"\uf1d6"}.fa-wechat:before,.fa-weixin:before{content:"\uf1d7"}.fa-send:before,.fa-paper-plane:before{content:"\uf1d8"}.fa-send-o:before,.fa-paper-plane-o:before{content:"\uf1d9"}.fa-history:before{content:"\uf1da"}.fa-circle-thin:before{content:"\uf1db"}.fa-header:before{content:"\uf1dc"}.fa-paragraph:before{content:"\uf1dd"}.fa-sliders:before{content:"\uf1de"}.fa-share
-alt:before{content:"\uf1e0"}.fa-share-alt-square:before{content:"\uf1e1"}.fa-bomb:before{content:"\uf1e2"}.fa-soccer-ball-o:before,.fa-futbol-o:before{content:"\uf1e3"}.fa-tty:before{content:"\uf1e4"}.fa-binoculars:before{content:"\uf1e5"}.fa-plug:before{content:"\uf1e6"}.fa-slideshare:before{content:"\uf1e7"}.fa-twitch:before{content:"\uf1e8"}.fa-yelp:before{content:"\uf1e9"}.fa-newspaper-o:before{content:"\uf1ea"}.fa-wifi:before{content:"\uf1eb"}.fa-calculator:before{content:"\uf1ec"}.fa-paypal:before{content:"\uf1ed"}.fa-google-wallet:before{content:"\uf1ee"}.fa-cc-visa:before{content:"\uf1f0"}.fa-cc-mastercard:before{content:"\uf1f1"}.fa-cc-discover:before{content:"\uf1f2"}.fa-cc-amex:before{content:"\uf1f3"}.fa-cc-paypal:before{content:"\uf1f4"}.fa-cc-stripe:before{content:"\uf1f5"}.fa-bell-slash:before{content:"\uf1f6"}.fa-bell-slash-o:before{content:"\uf1f7"}.fa-trash:before{content:"\uf1f8"}.fa-copyright:before{content:"\uf1f9"}.fa-at:before{content:"\uf1fa"}.fa-eyedropper:before{content:"\uf1fb"}.fa-paint-brush:before{content:"\uf1fc"}.fa-birthday-cake:be
fore{content:"\uf1fd"}.fa-area-chart:before{content:"\uf1fe"}.fa-pie-chart:before{content:"\uf200"}.fa-line-chart:before{content:"\uf201"}.fa-lastfm:before{content:"\uf202"}.fa-lastfm-square:before{content:"\uf203"}.fa-toggle-off:before{content:"\uf204"}.fa-toggle-on:before{content:"\uf205"}.fa-bicycle:before{content:"\uf206"}.fa-bus:before{content:"\uf207"}.fa-ioxhost:before{content:"\uf208"}.fa-angellist:before{content:"\uf209"}.fa-cc:before{content:"\uf20a"}.fa-shekel:before,.fa-sheqel:before,.fa-ils:before{content:"\uf20b"}.fa-meanpath:before{content:"\uf20c"}.fa,.wy-menu-vertical li span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li.current>a span.toctree-expand,.rst-content .admonition-title,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content dl dt .headerlink,.rst-content p.caption .headerlink,.rst-content tt.download span:first-child,.rst-content code.downl
oad span:first-child,.icon,.wy-dropdown .caret,.wy-inline-validate.wy-inline-validate-success .wy-input-context,.wy-inline-validate.wy-inline-validate-danger .wy-input-context,.wy-inline-validate.wy-inline-validate-warning .wy-input-context,.wy-inline-validate.wy-inline-validate-info .wy-input-context{font-family:inherit}.fa:before,.wy-menu-vertical li span.toctree-expand:before,.wy-menu-vertical li.on a span.toctree-expand:before,.wy-menu-vertical li.current>a span.toctree-expand:before,.rst-content .admonition-title:before,.rst-content h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3 .headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5 .headerlink:before,.rst-content h6 .headerlink:before,.rst-content dl dt .headerlink:before,.rst-content p.caption .headerlink:before,.rst-content tt.download span:first-child:before,.rst-content code.download span:first-child:before,.icon:before,.wy-dropdown .caret:before,.wy-inline-validate.wy-inline-valida
te-success .wy-input-context:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before{font-family:"FontAwesome";display:inline-block;font-style:normal;font-weight:normal;line-height:1;text-decoration:inherit}a .fa,a .wy-menu-vertical li span.toctree-expand,.wy-menu-vertical li a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li.current>a span.toctree-expand,a .rst-content .admonition-title,.rst-content a .admonition-title,a .rst-content h1 .headerlink,.rst-content h1 a .headerlink,a .rst-content h2 .headerlink,.rst-content h2 a .headerlink,a .rst-content h3 .headerlink,.rst-content h3 a .headerlink,a .rst-content h4 .headerlink,.rst-content h4 a .headerlink,a .rst-content h5 .headerlink,.rst-content h5 a .headerlink,a .rst-content h6 .headerlink,.rst-content h6 a .headerlink,a .rst-conte
nt dl dt .headerlink,.rst-content dl dt a .headerlink,a .rst-content p.caption .headerlink,.rst-content p.caption a .headerlink,a .rst-content tt.download span:first-child,.rst-content tt.download a span:first-child,a .rst-content code.download span:first-child,.rst-content code.download a span:first-child,a .icon{display:inline-block;text-decoration:inherit}.btn .fa,.btn .wy-menu-vertical li span.toctree-expand,.wy-menu-vertical li .btn span.toctree-expand,.btn .wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li.on a .btn span.toctree-expand,.btn .wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.current>a .btn span.toctree-expand,.btn .rst-content .admonition-title,.rst-content .btn .admonition-title,.btn .rst-content h1 .headerlink,.rst-content h1 .btn .headerlink,.btn .rst-content h2 .headerlink,.rst-content h2 .btn .headerlink,.btn .rst-content h3 .headerlink,.rst-content h3 .btn .headerlink,.btn .rst-content h4 .headerlink,.rst-content h4 .b
tn .headerlink,.btn .rst-content h5 .headerlink,.rst-content h5 .btn .headerlink,.btn .rst-content h6 .headerlink,.rst-content h6 .btn .headerlink,.btn .rst-content dl dt .headerlink,.rst-content dl dt .btn .headerlink,.btn .rst-content p.caption .headerlink,.rst-content p.caption .btn .headerlink,.btn .rst-content tt.download span:first-child,.rst-content tt.download .btn span:first-child,.btn .rst-content code.download span:first-child,.rst-content code.download .btn span:first-child,.btn .icon,.nav .fa,.nav .wy-menu-vertical li span.toctree-expand,.wy-menu-vertical li .nav span.toctree-expand,.nav .wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li.on a .nav span.toctree-expand,.nav .wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.current>a .nav span.toctree-expand,.nav .rst-content .admonition-title,.rst-content .nav .admonition-title,.nav .rst-content h1 .headerlink,.rst-content h1 .nav .headerlink,.nav .rst-content h2 .headerlink,.rst-cont
ent h2 .nav .headerlink,.nav .rst-content h3 .headerlink,.rst-content h3 .nav .headerlink,.nav .rst-content h4 .headerlink,.rst-content h4 .nav .headerlink,.nav .rst-content h5 .headerlink,.rst-content h5 .nav .headerlink,.nav .rst-content h6 .headerlink,.rst-content h6 .nav .headerlink,.nav .rst-content dl dt .headerlink,.rst-content dl dt .nav .headerlink,.nav .rst-content p.caption .headerlink,.rst-content p.caption .nav .headerlink,.nav .rst-content tt.download span:first-child,.rst-content tt.download .nav span:first-child,.nav .rst-content code.download span:first-child,.rst-content code.download .nav span:first-child,.nav .icon{display:inline}.btn .fa.fa-large,.btn .wy-menu-vertical li span.fa-large.toctree-expand,.wy-menu-vertical li .btn span.fa-large.toctree-expand,.btn .rst-content .fa-large.admonition-title,.rst-content .btn .fa-large.admonition-title,.btn .rst-content h1 .fa-large.headerlink,.rst-content h1 .btn .fa-large.headerlink,.btn .rst-content h2 .fa-large.header
link,.rst-content h2 .btn .fa-large.headerlink,.btn .rst-content h3 .fa-large.headerlink,.rst-content h3 .btn .fa-large.headerlink,.btn .rst-content h4 .fa-large.headerlink,.rst-content h4 .btn .fa-large.headerlink,.btn .rst-content h5 .fa-large.headerlink,.rst-content h5 .btn .fa-large.headerlink,.btn .rst-content h6 .fa-large.headerlink,.rst-content h6 .btn .fa-large.headerlink,.btn .rst-content dl dt .fa-large.headerlink,.rst-content dl dt .btn .fa-large.headerlink,.btn .rst-content p.caption .fa-large.headerlink,.rst-content p.caption .btn .fa-large.headerlink,.btn .rst-content tt.download span.fa-large:first-child,.rst-content tt.download .btn span.fa-large:first-child,.btn .rst-content code.download span.fa-large:first-child,.rst-content code.download .btn span.fa-large:first-child,.btn .fa-large.icon,.nav .fa.fa-large,.nav .wy-menu-vertical li span.fa-large.toctree-expand,.wy-menu-vertical li .nav span.fa-large.toctree-expand,.nav .rst-content .fa-large.admonition-title,.rst-
content .nav .fa-large.admonition-title,.nav .rst-content h1 .fa-large.headerlink,.rst-content h1 .nav .fa-large.headerlink,.nav .rst-content h2 .fa-large.headerlink,.rst-content h2 .nav .fa-large.headerlink,.nav .rst-content h3 .fa-large.headerlink,.rst-content h3 .nav .fa-large.headerlink,.nav .rst-content h4 .fa-large.headerlink,.rst-content h4 .nav .fa-large.headerlink,.nav .rst-content h5 .fa-large.headerlink,.rst-content h5 .nav .fa-large.headerlink,.nav .rst-content h6 .fa-large.headerlink,.rst-content h6 .nav .fa-large.headerlink,.nav .rst-content dl dt .fa-large.headerlink,.rst-content dl dt .nav .fa-large.headerlink,.nav .rst-content p.caption .fa-large.headerlink,.rst-content p.caption .nav .fa-large.headerlink,.nav .rst-content tt.download span.fa-large:first-child,.rst-content tt.download .nav span.fa-large:first-child,.nav .rst-content code.download span.fa-large:first-child,.rst-content code.download .nav span.fa-large:first-child,.nav .fa-large.icon{line-height:0.9em
}.btn .fa.fa-spin,.btn .wy-menu-vertical li span.fa-spin.toctree-expand,.wy-menu-vertical li .btn span.fa-spin.toctree-expand,.btn .rst-content .fa-spin.admonition-title,.rst-content .btn .fa-spin.admonition-title,.btn .rst-content h1 .fa-spin.headerlink,.rst-content h1 .btn .fa-spin.headerlink,.btn .rst-content h2 .fa-spin.headerlink,.rst-content h2 .btn .fa-spin.headerlink,.btn .rst-content h3 .fa-spin.headerlink,.rst-content h3 .btn .fa-spin.headerlink,.btn .rst-content h4 .fa-spin.headerlink,.rst-content h4 .btn .fa-spin.headerlink,.btn .rst-content h5 .fa-spin.headerlink,.rst-content h5 .btn .fa-spin.headerlink,.btn .rst-content h6 .fa-spin.headerlink,.rst-content h6 .btn .fa-spin.headerlink,.btn .rst-content dl dt .fa-spin.headerlink,.rst-content dl dt .btn .fa-spin.headerlink,.btn .rst-content p.caption .fa-spin.headerlink,.rst-content p.caption .btn .fa-spin.headerlink,.btn .rst-content tt.download span.fa-spin:first-child,.rst-content tt.download .btn span.fa-spin:first-chi
ld,.btn .rst-content code.download span.fa-spin:first-child,.rst-content code.download .btn span.fa-spin:first-child,.btn .fa-spin.icon,.nav .fa.fa-spin,.nav .wy-menu-vertical li span.fa-spin.toctree-expand,.wy-menu-vertical li .nav span.fa-spin.toctree-expand,.nav .rst-content .fa-spin.admonition-title,.rst-content .nav .fa-spin.admonition-title,.nav .rst-content h1 .fa-spin.headerlink,.rst-content h1 .nav .fa-spin.headerlink,.nav .rst-content h2 .fa-spin.headerlink,.rst-content h2 .nav .fa-spin.headerlink,.nav .rst-content h3 .fa-spin.headerlink,.rst-content h3 .nav .fa-spin.headerlink,.nav .rst-content h4 .fa-spin.headerlink,.rst-content h4 .nav .fa-spin.headerlink,.nav .rst-content h5 .fa-spin.headerlink,.rst-content h5 .nav .fa-spin.headerlink,.nav .rst-content h6 .fa-spin.headerlink,.rst-content h6 .nav .fa-spin.headerlink,.nav .rst-content dl dt .fa-spin.headerlink,.rst-content dl dt .nav .fa-spin.headerlink,.nav .rst-content p.caption .fa-spin.headerlink,.rst-content p.capti
on .nav .fa-spin.headerlink,.nav .rst-content tt.download span.fa-spin:first-child,.rst-content tt.download .nav span.fa-spin:first-child,.nav .rst-content code.download span.fa-spin:first-child,.rst-content code.download .nav span.fa-spin:first-child,.nav .fa-spin.icon{display:inline-block}.btn.fa:before,.wy-menu-vertical li span.btn.toctree-expand:before,.rst-content .btn.admonition-title:before,.rst-content h1 .btn.headerlink:before,.rst-content h2 .btn.headerlink:before,.rst-content h3 .btn.headerlink:before,.rst-content h4 .btn.headerlink:before,.rst-content h5 .btn.headerlink:before,.rst-content h6 .btn.headerlink:before,.rst-content dl dt .btn.headerlink:before,.rst-content p.caption .btn.headerlink:before,.rst-content tt.download span.btn:first-child:before,.rst-content code.download span.btn:first-child:before,.btn.icon:before{opacity:0.5;-webkit-transition:opacity 0.05s ease-in;-moz-transition:opacity 0.05s ease-in;transition:opacity 0.05s ease-in}.btn.fa:hover:before,.wy-
menu-vertical li span.btn.toctree-expand:hover:before,.rst-content .btn.admonition-title:hover:before,.rst-content h1 .btn.headerlink:hover:before,.rst-content h2 .btn.headerlink:hover:before,.rst-content h3 .btn.headerlink:hover:before,.rst-content h4 .btn.headerlink:hover:before,.rst-content h5 .btn.headerlink:hover:before,.rst-content h6 .btn.headerlink:hover:before,.rst-content dl dt .btn.headerlink:hover:before,.rst-content p.caption .btn.headerlink:hover:before,.rst-content tt.download span.btn:first-child:hover:before,.rst-content code.download span.btn:first-child:hover:before,.btn.icon:hover:before{opacity:1}.btn-mini .fa:before,.btn-mini .wy-menu-vertical li span.toctree-expand:before,.wy-menu-vertical li .btn-mini span.toctree-expand:before,.btn-mini .rst-content .admonition-title:before,.rst-content .btn-mini .admonition-title:before,.btn-mini .rst-content h1 .headerlink:before,.rst-content h1 .btn-mini .headerlink:before,.btn-mini .rst-content h2 .headerlink:before,.rst
-content h2 .btn-mini .headerlink:before,.btn-mini .rst-content h3 .headerlink:before,.rst-content h3 .btn-mini .headerlink:before,.btn-mini .rst-content h4 .headerlink:before,.rst-content h4 .btn-mini .headerlink:before,.btn-mini .rst-content h5 .headerlink:before,.rst-content h5 .btn-mini .headerlink:before,.btn-mini .rst-content h6 .headerlink:before,.rst-content h6 .btn-mini .headerlink:before,.btn-mini .rst-content dl dt .headerlink:before,.rst-content dl dt .btn-mini .headerlink:before,.btn-mini .rst-content p.caption .headerlink:before,.rst-content p.caption .btn-mini .headerlink:before,.btn-mini .rst-content tt.download span:first-child:before,.rst-content tt.download .btn-mini span:first-child:before,.btn-mini .rst-content code.download span:first-child:before,.rst-content code.download .btn-mini span:first-child:before,.btn-mini .icon:before{font-size:14px;vertical-align:-15%}.wy-alert,.rst-content .note,.rst-content .attention,.rst-content .caution,.rst-content .danger,.r
st-content .error,.rst-content .hint,.rst-content .important,.rst-content .tip,.rst-content .warning,.rst-content .seealso,.rst-content .admonition-todo{padding:12px;line-height:24px;margin-bottom:24px;background:#e7f2fa}.wy-alert-title,.rst-content .admonition-title{color:#fff;font-weight:bold;display:block;color:#fff;background:#6ab0de;margin:-12px;padding:6px 12px;margin-bottom:12px}.wy-alert.wy-alert-danger,.rst-content .wy-alert-danger.note,.rst-content .wy-alert-danger.attention,.rst-content .wy-alert-danger.caution,.rst-content .danger,.rst-content .error,.rst-content .wy-alert-danger.hint,.rst-content .wy-alert-danger.important,.rst-content .wy-alert-danger.tip,.rst-content .wy-alert-danger.warning,.rst-content .wy-alert-danger.seealso,.rst-content .wy-alert-danger.admonition-todo{background:#fdf3f2}.wy-alert.wy-alert-danger .wy-alert-title,.rst-content .wy-alert-danger.note .wy-alert-title,.rst-content .wy-alert-danger.attention .wy-alert-title,.rst-content .wy-alert-danger
.caution .wy-alert-title,.rst-content .danger .wy-alert-title,.rst-content .error .wy-alert-title,.rst-content .wy-alert-danger.hint .wy-alert-title,.rst-content .wy-alert-danger.important .wy-alert-title,.rst-content .wy-alert-danger.tip .wy-alert-title,.rst-content .wy-alert-danger.warning .wy-alert-title,.rst-content .wy-alert-danger.seealso .wy-alert-title,.rst-content .wy-alert-danger.admonition-todo .wy-alert-title,.wy-alert.wy-alert-danger .rst-content .admonition-title,.rst-content .wy-alert.wy-alert-danger .admonition-title,.rst-content .wy-alert-danger.note .admonition-title,.rst-content .wy-alert-danger.attention .admonition-title,.rst-content .wy-alert-danger.caution .admonition-title,.rst-content .danger .admonition-title,.rst-content .error .admonition-title,.rst-content .wy-alert-danger.hint .admonition-title,.rst-content .wy-alert-danger.important .admonition-title,.rst-content .wy-alert-danger.tip .admonition-title,.rst-content .wy-alert-danger.warning .admonition-t
itle,.rst-content .wy-alert-danger.seealso .admonition-title,.rst-content .wy-alert-danger.admonition-todo .admonition-title{background:#f29f97}.wy-alert.wy-alert-warning,.rst-content .wy-alert-warning.note,.rst-content .attention,.rst-content .caution,.rst-content .wy-alert-warning.danger,.rst-content .wy-alert-warning.error,.rst-content .wy-alert-warning.hint,.rst-content .wy-alert-warning.important,.rst-content .wy-alert-warning.tip,.rst-content .warning,.rst-content .wy-alert-warning.seealso,.rst-content .admonition-todo{background:#ffedcc}.wy-alert.wy-alert-warning .wy-alert-title,.rst-content .wy-alert-warning.note .wy-alert-title,.rst-content .attention .wy-alert-title,.rst-content .caution .wy-alert-title,.rst-content .wy-alert-warning.danger .wy-alert-title,.rst-content .wy-alert-warning.error .wy-alert-title,.rst-content .wy-alert-warning.hint .wy-alert-title,.rst-content .wy-alert-warning.important .wy-alert-title,.rst-content .wy-alert-warning.tip .wy-alert-title,.rst-co
ntent .warning .wy-alert-title,.rst-content .wy-alert-warning.seealso .wy-alert-title,.rst-content .admonition-todo .wy-alert-title,.wy-alert.wy-alert-warning .rst-content .admonition-title,.rst-content .wy-alert.wy-alert-warning .admonition-title,.rst-content .wy-alert-warning.note .admonition-title,.rst-content .attention .admonition-title,.rst-content .caution .admonition-title,.rst-content .wy-alert-warning.danger .admonition-title,.rst-content .wy-alert-warning.error .admonition-title,.rst-content .wy-alert-warning.hint .admonition-title,.rst-content .wy-alert-warning.important .admonition-title,.rst-content .wy-alert-warning.tip .admonition-title,.rst-content .warning .admonition-title,.rst-content .wy-alert-warning.seealso .admonition-title,.rst-content .admonition-todo .admonition-title{background:#f0b37e}.wy-alert.wy-alert-info,.rst-content .note,.rst-content .wy-alert-info.attention,.rst-content .wy-alert-info.caution,.rst-content .wy-alert-info.danger,.rst-content .wy-ale
rt-info.error,.rst-content .wy-alert-info.hint,.rst-content .wy-alert-info.important,.rst-content .wy-alert-info.tip,.rst-content .wy-alert-info.warning,.rst-content .seealso,.rst-content .wy-alert-info.admonition-todo{background:#e7f2fa}.wy-alert.wy-alert-info .wy-alert-title,.rst-content .note .wy-alert-title,.rst-content .wy-alert-info.attention .wy-alert-title,.rst-content .wy-alert-info.caution .wy-alert-title,.rst-content .wy-alert-info.danger .wy-alert-title,.rst-content .wy-alert-info.error .wy-alert-title,.rst-content .wy-alert-info.hint .wy-alert-title,.rst-content .wy-alert-info.important .wy-alert-title,.rst-content .wy-alert-info.tip .wy-alert-title,.rst-content .wy-alert-info.warning .wy-alert-title,.rst-content .seealso .wy-alert-title,.rst-content .wy-alert-info.admonition-todo .wy-alert-title,.wy-alert.wy-alert-info .rst-content .admonition-title,.rst-content .wy-alert.wy-alert-info .admonition-title,.rst-content .note .admonition-title,.rst-content .wy-alert-info.a
ttention .admonition-title,.rst-content .wy-alert-info.caution .admonition-title,.rst-content .wy-alert-info.danger .admonition-title,.rst-content .wy-alert-info.error .admonition-title,.rst-content .wy-alert-info.hint .admonition-title,.rst-content .wy-alert-info.important .admonition-title,.rst-content .wy-alert-info.tip .admonition-title,.rst-content .wy-alert-info.warning .admonition-title,.rst-content .seealso .admonition-title,.rst-content .wy-alert-info.admonition-todo .admonition-title{background:#6ab0de}.wy-alert.wy-alert-success,.rst-content .wy-alert-success.note,.rst-content .wy-alert-success.attention,.rst-content .wy-alert-success.caution,.rst-content .wy-alert-success.danger,.rst-content .wy-alert-success.error,.rst-content .hint,.rst-content .important,.rst-content .tip,.rst-content .wy-alert-success.warning,.rst-content .wy-alert-success.seealso,.rst-content .wy-alert-success.admonition-todo{background:#dbfaf4}.wy-alert.wy-alert-success .wy-alert-title,.rst-content
.wy-alert-success.note .wy-alert-title,.rst-content .wy-alert-success.attention .wy-alert-title,.rst-content .wy-alert-success.caution .wy-alert-title,.rst-content .wy-alert-success.danger .wy-alert-title,.rst-content .wy-alert-success.error .wy-alert-title,.rst-content .hint .wy-alert-title,.rst-content .important .wy-alert-title,.rst-content .tip .wy-alert-title,.rst-content .wy-alert-success.warning .wy-alert-title,.rst-content .wy-alert-success.seealso .wy-alert-title,.rst-content .wy-alert-success.admonition-todo .wy-alert-title,.wy-alert.wy-alert-success .rst-content .admonition-title,.rst-content .wy-alert.wy-alert-success .admonition-title,.rst-content .wy-alert-success.note .admonition-title,.rst-content .wy-alert-success.attention .admonition-title,.rst-content .wy-alert-success.caution .admonition-title,.rst-content .wy-alert-success.danger .admonition-title,.rst-content .wy-alert-success.error .admonition-title,.rst-content .hint .admonition-title,.rst-content .important
.admonition-title,.rst-content .tip .admonition-title,.rst-content .wy-alert-success.warning .admonition-title,.rst-content .wy-alert-success.seealso .admonition-title,.rst-content .wy-alert-success.admonition-todo .admonition-title{background:#1abc9c}.wy-alert.wy-alert-neutral,.rst-content .wy-alert-neutral.note,.rst-content .wy-alert-neutral.attention,.rst-content .wy-alert-neutral.caution,.rst-content .wy-alert-neutral.danger,.rst-content .wy-alert-neutral.error,.rst-content .wy-alert-neutral.hint,.rst-content .wy-alert-neutral.important,.rst-content .wy-alert-neutral.tip,.rst-content .wy-alert-neutral.warning,.rst-content .wy-alert-neutral.seealso,.rst-content .wy-alert-neutral.admonition-todo{background:#f3f6f6}.wy-alert.wy-alert-neutral .wy-alert-title,.rst-content .wy-alert-neutral.note .wy-alert-title,.rst-content .wy-alert-neutral.attention .wy-alert-title,.rst-content .wy-alert-neutral.caution .wy-alert-title,.rst-content .wy-alert-neutral.danger .wy-alert-title,.rst-cont
ent .wy-alert-neutral.error .wy-alert-title,.rst-content .wy-alert-neutral.hint .wy-alert-title,.rst-content .wy-alert-neutral.important .wy-alert-title,.rst-content .wy-alert-neutral.tip .wy-alert-title,.rst-content .wy-alert-neutral.warning .wy-alert-title,.rst-content .wy-alert-neutral.seealso .wy-alert-title,.rst-content .wy-alert-neutral.admonition-todo .wy-alert-title,.wy-alert.wy-alert-neutral .rst-content .admonition-title,.rst-content .wy-alert.wy-alert-neutral .admonition-title,.rst-content .wy-alert-neutral.note .admonition-title,.rst-content .wy-alert-neutral.attention .admonition-title,.rst-content .wy-alert-neutral.caution .admonition-title,.rst-content .wy-alert-neutral.danger .admonition-title,.rst-content .wy-alert-neutral.error .admonition-title,.rst-content .wy-alert-neutral.hint .admonition-title,.rst-content .wy-alert-neutral.important .admonition-title,.rst-content .wy-alert-neutral.tip .admonition-title,.rst-content .wy-alert-neutral.warning .admonition-title,
.rst-content .wy-alert-neutral.seealso .admonition-title,.rst-content .wy-alert-neutral.admonition-todo .admonition-title{color:#404040;background:#e1e4e5}.wy-alert.wy-alert-neutral a,.rst-content .wy-alert-neutral.note a,.rst-content .wy-alert-neutral.attention a,.rst-content .wy-alert-neutral.caution a,.rst-content .wy-alert-neutral.danger a,.rst-content .wy-alert-neutral.error a,.rst-content .wy-alert-neutral.hint a,.rst-content .wy-alert-neutral.important a,.rst-content .wy-alert-neutral.tip a,.rst-content .wy-alert-neutral.warning a,.rst-content .wy-alert-neutral.seealso a,.rst-content .wy-alert-neutral.admonition-todo a{color:#2980B9}.wy-alert p:last-child,.rst-content .note p:last-child,.rst-content .attention p:last-child,.rst-content .caution p:last-child,.rst-content .danger p:last-child,.rst-content .error p:last-child,.rst-content .hint p:last-child,.rst-content .important p:last-child,.rst-content .tip p:last-child,.rst-content .warning p:last-child,.rst-content .seeals
o p:last-child,.rst-content .admonition-todo p:last-child{margin-bottom:0}.wy-tray-container{position:fixed;bottom:0px;left:0;z-index:600}.wy-tray-container li{display:block;width:300px;background:transparent;color:#fff;text-align:center;box-shadow:0 5px 5px 0 rgba(0,0,0,0.1);padding:0 24px;min-width:20%;opacity:0;height:0;line-height:56px;overflow:hidden;-webkit-transition:all 0.3s ease-in;-moz-transition:all 0.3s ease-in;transition:all 0.3s ease-in}.wy-tray-container li.wy-tray-item-success{background:#27AE60}.wy-tray-container li.wy-tray-item-info{background:#2980B9}.wy-tray-container li.wy-tray-item-warning{background:#E67E22}.wy-tray-container li.wy-tray-item-danger{background:#E74C3C}.wy-tray-container li.on{opacity:1;height:56px}@media screen and (max-width: 768px){.wy-tray-container{bottom:auto;top:0;width:100%}.wy-tray-container li{width:100%}}button{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle;cursor:pointer;line-height:normal;-webkit-appearance:b
utton;*overflow:visible}button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}button[disabled]{cursor:default}.btn{display:inline-block;border-radius:2px;line-height:normal;white-space:nowrap;text-align:center;cursor:pointer;font-size:100%;padding:6px 12px 8px 12px;color:#fff;border:1px solid rgba(0,0,0,0.1);background-color:#27AE60;text-decoration:none;font-weight:normal;font-family:"Lato","proxima-nova","Helvetica Neue",Arial,sans-serif;box-shadow:0px 1px 2px -1px rgba(255,255,255,0.5) inset,0px -2px 0px 0px rgba(0,0,0,0.1) inset;outline-none:false;vertical-align:middle;*display:inline;zoom:1;-webkit-user-drag:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transition:all 0.1s linear;-moz-transition:all 0.1s linear;transition:all 0.1s linear}.btn-hover{background:#2e8ece;color:#fff}.btn:hover{background:#2cc36b;color:#fff}.btn:focus{background:#2cc36b;outline:0}.btn:active{box-shadow:0px -1px 0px 0px rgba(0,0,0,0.05)
inset,0px 2px 0px 0px rgba(0,0,0,0.1) inset;padding:8px 12px 6px 12px}.btn:visited{color:#fff}.btn:disabled{background-image:none;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);filter:alpha(opacity=40);opacity:0.4;cursor:not-allowed;box-shadow:none}.btn-disabled{background-image:none;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);filter:alpha(opacity=40);opacity:0.4;cursor:not-allowed;box-shadow:none}.btn-disabled:hover,.btn-disabled:focus,.btn-disabled:active{background-image:none;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);filter:alpha(opacity=40);opacity:0.4;cursor:not-allowed;box-shadow:none}.btn::-moz-focus-inner{padding:0;border:0}.btn-small{font-size:80%}.btn-info{background-color:#2980B9 !important}.btn-info:hover{background-color:#2e8ece !important}.btn-neutral{background-color:#f3f6f6 !important;color:#404040 !important}.btn-neutral:hover{background-color:#e5ebeb !important;color:#404040}.btn-neutral:visited{co
lor:#404040 !important}.btn-success{background-color:#27AE60 !important}.btn-success:hover{background-color:#295 !important}.btn-danger{background-color:#E74C3C !important}.btn-danger:hover{background-color:#ea6153 !important}.btn-warning{background-color:#E67E22 !important}.btn-warning:hover{background-color:#e98b39 !important}.btn-invert{background-color:#222}.btn-invert:hover{background-color:#2f2f2f !important}.btn-link{background-color:transparent !important;color:#2980B9;box-shadow:none;border-color:transparent !important}.btn-link:hover{background-color:transparent !important;color:#409ad5 !important;box-shadow:none}.btn-link:active{background-color:transparent !important;color:#409ad5 !important;box-shadow:none}.btn-link:visited{color:#9B59B6}.wy-btn-group .btn,.wy-control .btn{vertical-align:middle}.wy-btn-group{margin-bottom:24px;*zoom:1}.wy-btn-group:before,.wy-btn-group:after{display:table;content:""}.wy-btn-group:after{clear:both}.wy-dropdown{position:relative;display:i
nline-block}.wy-dropdown-active .wy-dropdown-menu{display:block}.wy-dropdown-menu{position:absolute;left:0;display:none;float:left;top:100%;min-width:100%;background:#fcfcfc;z-index:100;border:solid 1px #cfd7dd;box-shadow:0 2px 2px 0 rgba(0,0,0,0.1);padding:12px}.wy-dropdown-menu>dd>a{display:block;clear:both;color:#404040;white-space:nowrap;font-size:90%;padding:0 12px;cursor:pointer}.wy-dropdown-menu>dd>a:hover{background:#2980B9;color:#fff}.wy-dropdown-menu>dd.divider{border-top:solid 1px #cfd7dd;margin:6px 0}.wy-dropdown-menu>dd.search{padding-bottom:12px}.wy-dropdown-menu>dd.search input[type="search"]{width:100%}.wy-dropdown-menu>dd.call-to-action{background:#e3e3e3;text-transform:uppercase;font-weight:500;font-size:80%}.wy-dropdown-menu>dd.call-to-action:hover{background:#e3e3e3}.wy-dropdown-menu>dd.call-to-action .btn{color:#fff}.wy-dropdown.wy-dropdown-up .wy-dropdown-menu{bottom:100%;top:auto;left:auto;right:0}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu{background:#f
cfcfc;margin-top:2px}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu a{padding:6px 12px}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu a:hover{background:#2980B9;color:#fff}.wy-dropdown.wy-dropdown-left .wy-dropdown-menu{right:0;left:auto;text-align:right}.wy-dropdown-arrow:before{content:" ";border-bottom:5px solid #f5f5f5;border-left:5px solid transparent;border-right:5px solid transparent;position:absolute;display:block;top:-4px;left:50%;margin-left:-3px}.wy-dropdown-arrow.wy-dropdown-arrow-left:before{left:11px}.wy-form-stacked select{display:block}.wy-form-aligned input,.wy-form-aligned textarea,.wy-form-aligned select,.wy-form-aligned .wy-help-inline,.wy-form-aligned label{display:inline-block;*display:inline;*zoom:1;vertical-align:middle}.wy-form-aligned .wy-control-group>label{display:inline-block;vertical-align:middle;width:10em;margin:6px 12px 0 0;float:left}.wy-form-aligned .wy-control{float:left}.wy-form-aligned .wy-control label{display:block}.wy-form-aligned .wy-c
ontrol select{margin-top:6px}fieldset{border:0;margin:0;padding:0}legend{display:block;width:100%;border:0;padding:0;white-space:normal;margin-bottom:24px;font-size:150%;*margin-left:-7px}label{display:block;margin:0 0 0.3125em 0;color:#333;font-size:90%}input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}.wy-control-group{margin-bottom:24px;*zoom:1;max-width:68em;margin-left:auto;margin-right:auto;*zoom:1}.wy-control-group:before,.wy-control-group:after{display:table;content:""}.wy-control-group:after{clear:both}.wy-control-group:before,.wy-control-group:after{display:table;content:""}.wy-control-group:after{clear:both}.wy-control-group.wy-control-group-required>label:after{content:" *";color:#E74C3C}.wy-control-group .wy-form-full,.wy-control-group .wy-form-halves,.wy-control-group .wy-form-thirds{padding-bottom:12px}.wy-control-group .wy-form-full select,.wy-control-group .wy-form-halves select,.wy-control-group .wy-form-thirds select{widt
h:100%}.wy-control-group .wy-form-full input[type="text"],.wy-control-group .wy-form-full input[type="password"],.wy-control-group .wy-form-full input[type="email"],.wy-control-group .wy-form-full input[type="url"],.wy-control-group .wy-form-full input[type="date"],.wy-control-group .wy-form-full input[type="month"],.wy-control-group .wy-form-full input[type="time"],.wy-control-group .wy-form-full input[type="datetime"],.wy-control-group .wy-form-full input[type="datetime-local"],.wy-control-group .wy-form-full input[type="week"],.wy-control-group .wy-form-full input[type="number"],.wy-control-group .wy-form-full input[type="search"],.wy-control-group .wy-form-full input[type="tel"],.wy-control-group .wy-form-full input[type="color"],.wy-control-group .wy-form-halves input[type="text"],.wy-control-group .wy-form-halves input[type="password"],.wy-control-group .wy-form-halves input[type="email"],.wy-control-group .wy-form-halves input[type="url"],.wy-control-group .wy-form-halves inp
ut[type="date"],.wy-control-group .wy-form-halves input[type="month"],.wy-control-group .wy-form-halves input[type="time"],.wy-control-group .wy-form-halves input[type="datetime"],.wy-control-group .wy-form-halves input[type="datetime-local"],.wy-control-group .wy-form-halves input[type="week"],.wy-control-group .wy-form-halves input[type="number"],.wy-control-group .wy-form-halves input[type="search"],.wy-control-group .wy-form-halves input[type="tel"],.wy-control-group .wy-form-halves input[type="color"],.wy-control-group .wy-form-thirds input[type="text"],.wy-control-group .wy-form-thirds input[type="password"],.wy-control-group .wy-form-thirds input[type="email"],.wy-control-group .wy-form-thirds input[type="url"],.wy-control-group .wy-form-thirds input[type="date"],.wy-control-group .wy-form-thirds input[type="month"],.wy-control-group .wy-form-thirds input[type="time"],.wy-control-group .wy-form-thirds input[type="datetime"],.wy-control-group .wy-form-thirds input[type="dateti
me-local"],.wy-control-group .wy-form-thirds input[type="week"],.wy-control-group .wy-form-thirds input[type="number"],.wy-control-group .wy-form-thirds input[type="search"],.wy-control-group .wy-form-thirds input[type="tel"],.wy-control-group .wy-form-thirds input[type="color"]{width:100%}.wy-control-group .wy-form-full{float:left;display:block;margin-right:2.35765%;width:100%;margin-right:0}.wy-control-group .wy-form-full:last-child{margin-right:0}.wy-control-group .wy-form-halves{float:left;display:block;margin-right:2.35765%;width:48.82117%}.wy-control-group .wy-form-halves:last-child{margin-right:0}.wy-control-group .wy-form-halves:nth-of-type(2n){margin-right:0}.wy-control-group .wy-form-halves:nth-of-type(2n+1){clear:left}.wy-control-group .wy-form-thirds{float:left;display:block;margin-right:2.35765%;width:31.76157%}.wy-control-group .wy-form-thirds:last-child{margin-right:0}.wy-control-group .wy-form-thirds:nth-of-type(3n){margin-right:0}.wy-control-group .wy-form-thirds:nt
h-of-type(3n+1){clear:left}.wy-control-group.wy-control-group-no-input .wy-control{margin:6px 0 0 0;font-size:90%}.wy-control-no-input{display:inline-block;margin:6px 0 0 0;font-size:90%}.wy-control-group.fluid-input input[type="text"],.wy-control-group.fluid-input input[type="password"],.wy-control-group.fluid-input input[type="email"],.wy-control-group.fluid-input input[type="url"],.wy-control-group.fluid-input input[type="date"],.wy-control-group.fluid-input input[type="month"],.wy-control-group.fluid-input input[type="time"],.wy-control-group.fluid-input input[type="datetime"],.wy-control-group.fluid-input input[type="datetime-local"],.wy-control-group.fluid-input input[type="week"],.wy-control-group.fluid-input input[type="number"],.wy-control-group.fluid-input input[type="search"],.wy-control-group.fluid-input input[type="tel"],.wy-control-group.fluid-input input[type="color"]{width:100%}.wy-form-message-inline{display:inline-block;padding-left:0.3em;color:#666;vertical-align:
middle;font-size:90%}.wy-form-message{display:block;color:#999;font-size:70%;margin-top:0.3125em;font-style:italic}.wy-form-message p{font-size:inherit;font-style:italic;margin-bottom:6px}.wy-form-message p:last-child{margin-bottom:0}input{line-height:normal}input[type="button"],input[type="reset"],input[type="submit"]{-webkit-appearance:button;cursor:pointer;font-family:"Lato","proxima-nova","Helvetica Neue",Arial,sans-serif;*overflow:visible}input[type="text"],input[type="password"],input[type="email"],input[type="url"],input[type="date"],input[type="month"],input[type="time"],input[type="datetime"],input[type="datetime-local"],input[type="week"],input[type="number"],input[type="search"],input[type="tel"],input[type="color"]{-webkit-appearance:none;padding:6px;display:inline-block;border:1px solid #ccc;font-size:80%;font-family:"Lato","proxima-nova","Helvetica Neue",Arial,sans-serif;box-shadow:inset 0 1px 3px #ddd;border-radius:0;-webkit-transition:border 0.3s linear;-moz-transiti
on:border 0.3s linear;transition:border 0.3s linear}input[type="datetime-local"]{padding:0.34375em 0.625em}input[disabled]{cursor:default}input[type="checkbox"],input[type="radio"]{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box;padding:0;margin-right:0.3125em;*height:13px;*width:13px}input[type="search"]{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}input[type="search"]::-webkit-search-cancel-button,input[type="search"]::-webkit-search-decoration{-webkit-appearance:none}input[type="text"]:focus,input[type="password"]:focus,input[type="email"]:focus,input[type="url"]:focus,input[type="date"]:focus,input[type="month"]:focus,input[type="time"]:focus,input[type="datetime"]:focus,input[type="datetime-local"]:focus,input[type="week"]:focus,input[type="number"]:focus,input[type="search"]:focus,input[type="tel"]:focus,input[type="color"]:focus{outline:0;outline:thin dotted \9;border-color:#333}input.no-focus:focus{border-color
:#ccc !important}input[type="file"]:focus,input[type="radio"]:focus,input[type="checkbox"]:focus{outline:thin dotted #333;outline:1px auto #129FEA}input[type="text"][disabled],input[type="password"][disabled],input[type="email"][disabled],input[type="url"][disabled],input[type="date"][disabled],input[type="month"][disabled],input[type="time"][disabled],input[type="datetime"][disabled],input[type="datetime-local"][disabled],input[type="week"][disabled],input[type="number"][disabled],input[type="search"][disabled],input[type="tel"][disabled],input[type="color"][disabled]{cursor:not-allowed;background-color:#fafafa}input:focus:invalid,textarea:focus:invalid,select:focus:invalid{color:#E74C3C;border:1px solid #E74C3C}input:focus:invalid:focus,textarea:focus:invalid:focus,select:focus:invalid:focus{border-color:#E74C3C}input[type="file"]:focus:invalid:focus,input[type="radio"]:focus:invalid:focus,input[type="checkbox"]:focus:invalid:focus{outline-color:#E74C3C}input.wy-input-large{paddin
g:12px;font-size:100%}textarea{overflow:auto;vertical-align:top;width:100%;font-family:"Lato","proxima-nova","Helvetica Neue",Arial,sans-serif}select,textarea{padding:0.5em 0.625em;display:inline-block;border:1px solid #ccc;font-size:80%;box-shadow:inset 0 1px 3px #ddd;-webkit-transition:border 0.3s linear;-moz-transition:border 0.3s linear;transition:border 0.3s linear}select{border:1px solid #ccc;background-color:#fff}select[multiple]{height:auto}select:focus,textarea:focus{outline:0}select[disabled],textarea[disabled],input[readonly],select[readonly],textarea[readonly]{cursor:not-allowed;background-color:#fafafa}input[type="radio"][disabled],input[type="checkbox"][disabled]{cursor:not-allowed}.wy-checkbox,.wy-radio{margin:6px 0;color:#404040;display:block}.wy-checkbox input,.wy-radio input{vertical-align:baseline}.wy-form-message-inline{display:inline-block;*display:inline;*zoom:1;vertical-align:middle}.wy-input-prefix,.wy-input-suffix{white-space:nowrap;padding:6px}.wy-input-pre
fix .wy-input-context,.wy-input-suffix .wy-input-context{line-height:27px;padding:0 8px;display:inline-block;font-size:80%;background-color:#f3f6f6;border:solid 1px #ccc;color:#999}.wy-input-suffix .wy-input-context{border-left:0}.wy-input-prefix .wy-input-context{border-right:0}.wy-switch{width:36px;height:12px;margin:12px 0;position:relative;border-radius:4px;background:#ccc;cursor:pointer;-webkit-transition:all 0.2s ease-in-out;-moz-transition:all 0.2s ease-in-out;transition:all 0.2s ease-in-out}.wy-switch:before{position:absolute;content:"";display:block;width:18px;height:18px;border-radius:4px;background:#999;left:-3px;top:-3px;-webkit-transition:all 0.2s ease-in-out;-moz-transition:all 0.2s ease-in-out;transition:all 0.2s ease-in-out}.wy-switch:after{content:"false";position:absolute;left:48px;display:block;font-size:12px;color:#ccc}.wy-switch.active{background:#1e8449}.wy-switch.active:before{left:24px;background:#27AE60}.wy-switch.active:after{content:"true"}.wy-switch.disab
led,.wy-switch.active.disabled{cursor:not-allowed}.wy-control-group.wy-control-group-error .wy-form-message,.wy-control-group.wy-control-group-error>label{color:#E74C3C}.wy-control-group.wy-control-group-error input[type="text"],.wy-control-group.wy-control-group-error input[type="password"],.wy-control-group.wy-control-group-error input[type="email"],.wy-control-group.wy-control-group-error input[type="url"],.wy-control-group.wy-control-group-error input[type="date"],.wy-control-group.wy-control-group-error input[type="month"],.wy-control-group.wy-control-group-error input[type="time"],.wy-control-group.wy-control-group-error input[type="datetime"],.wy-control-group.wy-control-group-error input[type="datetime-local"],.wy-control-group.wy-control-group-error input[type="week"],.wy-control-group.wy-control-group-error input[type="number"],.wy-control-group.wy-control-group-error input[type="search"],.wy-control-group.wy-control-group-error input[type="tel"],.wy-control-group.wy-contr
ol-group-error input[type="color"]{border:solid 1px #E74C3C}.wy-control-group.wy-control-group-error textarea{border:solid 1px #E74C3C}.wy-inline-validate{white-space:nowrap}.wy-inline-validate .wy-input-context{padding:0.5em 0.625em;display:inline-block;font-size:80%}.wy-inline-validate.wy-inline-validate-success .wy-input-context{color:#27AE60}.wy-inline-validate.wy-inline-validate-danger .wy-input-context{color:#E74C3C}.wy-inline-validate.wy-inline-validate-warning .wy-input-context{color:#E67E22}.wy-inline-validate.wy-inline-validate-info .wy-input-context{color:#2980B9}.rotate-90{-webkit-transform:rotate(90deg);-moz-transform:rotate(90deg);-ms-transform:rotate(90deg);-o-transform:rotate(90deg);transform:rotate(90deg)}.rotate-180{-webkit-transform:rotate(180deg);-moz-transform:rotate(180deg);-ms-transform:rotate(180deg);-o-transform:rotate(180deg);transform:rotate(180deg)}.rotate-270{-webkit-transform:rotate(270deg);-moz-transform:rotate(270deg);-ms-transform:rotate(270deg);-o-t
ransform:rotate(270deg);transform:rotate(270deg)}.mirror{-webkit-transform:scaleX(-1);-moz-transform:scaleX(-1);-ms-transform:scaleX(-1);-o-transform:scaleX(-1);transform:scaleX(-1)}.mirror.rotate-90{-webkit-transform:scaleX(-1) rotate(90deg);-moz-transform:scaleX(-1) rotate(90deg);-ms-transform:scaleX(-1) rotate(90deg);-o-transform:scaleX(-1) rotate(90deg);transform:scaleX(-1) rotate(90deg)}.mirror.rotate-180{-webkit-transform:scaleX(-1) rotate(180deg);-moz-transform:scaleX(-1) rotate(180deg);-ms-transform:scaleX(-1) rotate(180deg);-o-transform:scaleX(-1) rotate(180deg);transform:scaleX(-1) rotate(180deg)}.mirror.rotate-270{-webkit-transform:scaleX(-1) rotate(270deg);-moz-transform:scaleX(-1) rotate(270deg);-ms-transform:scaleX(-1) rotate(270deg);-o-transform:scaleX(-1) rotate(270deg);transform:scaleX(-1) rotate(270deg)}@media only screen and (max-width: 480px){.wy-form button[type="submit"]{margin:0.7em 0 0}.wy-form input[type="text"],.wy-form input[type="password"],.wy-form input
[type="email"],.wy-form input[type="url"],.wy-form input[type="date"],.wy-form input[type="month"],.wy-form input[type="time"],.wy-form input[type="datetime"],.wy-form input[type="datetime-local"],.wy-form input[type="week"],.wy-form input[type="number"],.wy-form input[type="search"],.wy-form input[type="tel"],.wy-form input[type="color"]{margin-bottom:0.3em;display:block}.wy-form label{margin-bottom:0.3em;display:block}.wy-form input[type="password"],.wy-form input[type="email"],.wy-form input[type="url"],.wy-form input[type="date"],.wy-form input[type="month"],.wy-form input[type="time"],.wy-form input[type="datetime"],.wy-form input[type="datetime-local"],.wy-form input[type="week"],.wy-form input[type="number"],.wy-form input[type="search"],.wy-form input[type="tel"],.wy-form input[type="color"]{margin-bottom:0}.wy-form-aligned .wy-control-group label{margin-bottom:0.3em;text-align:left;display:block;width:100%}.wy-form-aligned .wy-control{margin:1.5em 0 0 0}.wy-form .wy-help-in
line,.wy-form-message-inline,.wy-form-message{display:block;font-size:80%;padding:6px 0}}@media screen and (max-width: 768px){.tablet-hide{display:none}}@media screen and (max-width: 480px){.mobile-hide{display:none}}.float-left{float:left}.float-right{float:right}.full-width{width:100%}.wy-table,.rst-content table.docutils,.rst-content table.field-list{border-collapse:collapse;border-spacing:0;empty-cells:show;margin-bottom:24px}.wy-table caption,.rst-content table.docutils caption,.rst-content table.field-list caption{color:#000;font:italic 85%/1 arial,sans-serif;padding:1em 0;text-align:center}.wy-table td,.rst-content table.docutils td,.rst-content table.field-list td,.wy-table th,.rst-content table.docutils th,.rst-content table.field-list th{font-size:90%;margin:0;overflow:visible;padding:8px 16px}.wy-table td:first-child,.rst-content table.docutils td:first-child,.rst-content table.field-list td:first-child,.wy-table th:first-child,.rst-content table.docutils th:first-child,.
rst-content table.field-list th:first-child{border-left-width:0}.wy-table thead,.rst-content table.docutils thead,.rst-content table.field-list thead{color:#000;text-align:left;vertical-align:bottom;white-space:nowrap}.wy-table thead th,.rst-content table.docutils thead th,.rst-content table.field-list thead th{font-weight:bold;border-bottom:solid 2px #e1e4e5}.wy-table td,.rst-content table.docutils td,.rst-content table.field-list td{background-color:transparent;vertical-align:middle}.wy-table td p,.rst-content table.docutils td p,.rst-content table.field-list td p{line-height:18px}.wy-table td p:last-child,.rst-content table.docutils td p:last-child,.rst-content table.field-list td p:last-child{margin-bottom:0}.wy-table .wy-table-cell-min,.rst-content table.docutils .wy-table-cell-min,.rst-content table.field-list .wy-table-cell-min{width:1%;padding-right:0}.wy-table .wy-table-cell-min input[type=checkbox],.rst-content table.docutils .wy-table-cell-min input[type=checkbox],.rst-co
ntent table.field-list .wy-table-cell-min input[type=checkbox],.wy-table .wy-table-cell-min input[type=checkbox],.rst-content table.docutils .wy-table-cell-min input[type=checkbox],.rst-content table.field-list .wy-table-cell-min input[type=checkbox]{margin:0}.wy-table-secondary{color:gray;font-size:90%}.wy-table-tertiary{color:gray;font-size:80%}.wy-table-odd td,.wy-table-striped tr:nth-child(2n-1) td,.rst-content table.docutils:not(.field-list) tr:nth-child(2n-1) td{background-color:#f3f6f6}.wy-table-backed{background-color:#f3f6f6}.wy-table-bordered-all,.rst-content table.docutils{border:1px solid #e1e4e5}.wy-table-bordered-all td,.rst-content table.docutils td{border-bottom:1px solid #e1e4e5;border-left:1px solid #e1e4e5}.wy-table-bordered-all tbody>tr:last-child td,.rst-content table.docutils tbody>tr:last-child td{border-bottom-width:0}.wy-table-bordered{border:1px solid #e1e4e5}.wy-table-bordered-rows td{border-bottom:1px solid #e1e4e5}.wy-table-bordered-rows tbody>tr:last-ch
ild td{border-bottom-width:0}.wy-table-horizontal tbody>tr:last-child td{border-bottom-width:0}.wy-table-horizontal td,.wy-table-horizontal th{border-width:0 0 1px 0;border-bottom:1px solid #e1e4e5}.wy-table-horizontal tbody>tr:last-child td{border-bottom-width:0}.wy-table-responsive{margin-bottom:24px;max-width:100%;overflow:auto}.wy-table-responsive table{margin-bottom:0 !important}.wy-table-responsive table td,.wy-table-responsive table th{white-space:nowrap}a{color:#2980B9;text-decoration:none;cursor:pointer}a:hover{color:#3091d1}a:visited{color:#9B59B6}html{height:100%;overflow-x:hidden}body{font-family:"Lato","proxima-nova","Helvetica Neue",Arial,sans-serif;font-weight:normal;color:#404040;min-height:100%;overflow-x:hidden;background:#edf0f2}.wy-text-left{text-align:left}.wy-text-center{text-align:center}.wy-text-right{text-align:right}.wy-text-large{font-size:120%}.wy-text-normal{font-size:100%}.wy-text-small,small{font-size:80%}.wy-text-strike{text-decoration:line-through}.w
y-text-warning{color:#E67E22 !important}a.wy-text-warning:hover{color:#eb9950 !important}.wy-text-info{color:#2980B9 !important}a.wy-text-info:hover{color:#409ad5 !important}.wy-text-success{color:#27AE60 !important}a.wy-text-success:hover{color:#36d278 !important}.wy-text-danger{color:#E74C3C !important}a.wy-text-danger:hover{color:#ed7669 !important}.wy-text-neutral{color:#404040 !important}a.wy-text-neutral:hover{color:#595959 !important}h1,h2,.rst-content .toctree-wrapper p.caption,h3,h4,h5,h6,legend{margin-top:0;font-weight:700;font-family:"Roboto Slab","ff-tisa-web-pro","Georgia",Arial,sans-serif}p{line-height:24px;margin:0;font-size:16px;margin-bottom:24px}h1{font-size:175%}h2,.rst-content .toctree-wrapper p.caption{font-size:150%}h3{font-size:125%}h4{font-size:115%}h5{font-size:110%}h6{font-size:100%}hr{display:block;height:1px;border:0;border-top:1px solid #e1e4e5;margin:24px 0;padding:0}code,.rst-content tt,.rst-content code{white-space:nowrap;max-width:100%;background:#ff
f;border:solid 1px #e1e4e5;font-size:75%;padding:0 5px;font-family:Consolas,"Andale Mono WT","Andale Mono","Lucida Console","Lucida Sans Typewriter","DejaVu Sans Mono","Bitstream Vera Sans Mono","Liberation Mono","Nimbus Mono L",Monaco,"Courier New",Courier,monospace;color:#E74C3C;overflow-x:auto}code.code-large,.rst-content tt.code-large{font-size:90%}.wy-plain-list-disc,.rst-content .section ul,.rst-content .toctree-wrapper ul,article ul{list-style:disc;line-height:24px;margin-bottom:24px}.wy-plain-list-disc li,.rst-content .section ul li,.rst-content .toctree-wrapper ul li,article ul li{list-style:disc;margin-left:24px}.wy-plain-list-disc li p:last-child,.rst-content .section ul li p:last-child,.rst-content .toctree-wrapper ul li p:last-child,article ul li p:last-child{margin-bottom:0}.wy-plain-list-disc li ul,.rst-content .section ul li ul,.rst-content .toctree-wrapper ul li ul,article ul li ul{margin-bottom:0}.wy-plain-list-disc li li,.rst-content .section ul li li,.rst-content
.toctree-wrapper ul li li,article ul li li{list-style:circle}.wy-plain-list-disc li li li,.rst-content .section ul li li li,.rst-content .toctree-wrapper ul li li li,article ul li li li{list-style:square}.wy-plain-list-disc li ol li,.rst-content .section ul li ol li,.rst-content .toctree-wrapper ul li ol li,article ul li ol li{list-style:decimal}.wy-plain-list-decimal,.rst-content .section ol,.rst-content ol.arabic,article ol{list-style:decimal;line-height:24px;margin-bottom:24px}.wy-plain-list-decimal li,.rst-content .section ol li,.rst-content ol.arabic li,article ol li{list-style:decimal;margin-left:24px}.wy-plain-list-decimal li p:last-child,.rst-content .section ol li p:last-child,.rst-content ol.arabic li p:last-child,article ol li p:last-child{margin-bottom:0}.wy-plain-list-decimal li ul,.rst-content .section ol li ul,.rst-content ol.arabic li ul,article ol li ul{margin-bottom:0}.wy-plain-list-decimal li ul li,.rst-content .section ol li ul li,.rst-content ol.arabic li ul li
,article ol li ul li{list-style:disc}.codeblock-example{border:1px solid #e1e4e5;border-bottom:none;padding:24px;padding-top:48px;font-weight:500;background:#fff;position:relative}.codeblock-example:after{content:"Example";position:absolute;top:0px;left:0px;background:#9B59B6;color:#fff;padding:6px 12px}.codeblock-example.prettyprint-example-only{border:1px solid #e1e4e5;margin-bottom:24px}.codeblock,pre.literal-block,.rst-content .literal-block,.rst-content pre.literal-block,div[class^='highlight']{border:1px solid #e1e4e5;padding:0px;overflow-x:auto;background:#fff;margin:1px 0 24px 0}.codeblock div[class^='highlight'],pre.literal-block div[class^='highlight'],.rst-content .literal-block div[class^='highlight'],div[class^='highlight'] div[class^='highlight']{border:none;background:none;margin:0}div[class^='highlight'] td.code{width:100%}.linenodiv pre{border-right:solid 1px #e6e9ea;margin:0;padding:12px 12px;font-family:Consolas,"Andale Mono WT","Andale Mono","Lucida Console","Luc
ida Sans Typewriter","DejaVu Sans Mono","Bitstream Vera Sans Mono","Liberation Mono","Nimbus Mono L",Monaco,"Courier New",Courier,monospace;font-size:12px;line-height:1.5;color:#d9d9d9}div[class^='highlight'] pre{white-space:pre;margin:0;padding:12px 12px;font-family:Consolas,"Andale Mono WT","Andale Mono","Lucida Console","Lucida Sans Typewriter","DejaVu Sans Mono","Bitstream Vera Sans Mono","Liberation Mono","Nimbus Mono L",Monaco,"Courier New",Courier,monospace;font-size:12px;line-height:1.5;display:block;overflow:auto;color:#404040}@media print{.codeblock,pre.literal-block,.rst-content .literal-block,.rst-content pre.literal-block,div[class^='highlight'],div[class^='highlight'] pre{white-space:pre-wrap}}.hll{background-color:#ffc;margin:0 -12px;padding:0 12px;display:block}.c{color:#998;font-style:italic}.err{color:#a61717;background-color:#e3d2d2}.k{font-weight:bold}.o{font-weight:bold}.cm{color:#998;font-style:italic}.cp{color:#999;font-weight:bold}.c1{color:#998;font-style:it
alic}.cs{color:#999;font-weight:bold;font-style:italic}.gd{color:#000;background-color:#fdd}.gd .x{color:#000;background-color:#faa}.ge{font-style:italic}.gr{color:#a00}.gh{color:#999}.gi{color:#000;background-color:#dfd}.gi .x{color:#000;background-color:#afa}.go{color:#888}.gp{color:#555}.gs{font-weight:bold}.gu{color:purple;font-weight:bold}.gt{color:#a00}.kc{font-weight:bold}.kd{font-weight:bold}.kn{font-weight:bold}.kp{font-weight:bold}.kr{font-weight:bold}.kt{color:#458;font-weight:bold}.m{color:#099}.s{color:#d14}.n{color:#333}.na{color:teal}.nb{color:#0086b3}.nc{color:#458;font-weight:bold}.no{color:teal}.ni{color:purple}.ne{color:#900;font-weight:bold}.nf{color:#900;font-weight:bold}.nn{color:#555}.nt{color:navy}.nv{color:teal}.ow{font-weight:bold}.w{color:#bbb}.mf{color:#099}.mh{color:#099}.mi{color:#099}.mo{color:#099}.sb{color:#d14}.sc{color:#d14}.sd{color:#d14}.s2{color:#d14}.se{color:#d14}.sh{color:#d14}.si{color:#d14}.sx{color:#d14}.sr{color:#009926}.s1{color:#d14}.ss
{color:#990073}.bp{color:#999}.vc{color:teal}.vg{color:teal}.vi{color:teal}.il{color:#099}.gc{color:#999;background-color:#EAF2F5}.wy-breadcrumbs li{display:inline-block}.wy-breadcrumbs li.wy-breadcrumbs-aside{float:right}.wy-breadcrumbs li a{display:inline-block;padding:5px}.wy-breadcrumbs li a:first-child{padding-left:0}.wy-breadcrumbs li code,.wy-breadcrumbs li .rst-content tt,.rst-content .wy-breadcrumbs li tt{padding:5px;border:none;background:none}.wy-breadcrumbs li code.literal,.wy-breadcrumbs li .rst-content tt.literal,.rst-content .wy-breadcrumbs li tt.literal{color:#404040}.wy-breadcrumbs-extra{margin-bottom:0;color:#b3b3b3;font-size:80%;display:inline-block}@media screen and (max-width: 480px){.wy-breadcrumbs-extra{display:none}.wy-breadcrumbs li.wy-breadcrumbs-aside{display:none}}@media print{.wy-breadcrumbs li.wy-breadcrumbs-aside{display:none}}.wy-affix{position:fixed;top:1.618em}.wy-menu a:hover{text-decoration:none}.wy-menu-horiz{*zoom:1}.wy-menu-horiz:before,.wy-men
u-horiz:after{display:table;content:""}.wy-menu-horiz:after{clear:both}.wy-menu-horiz ul,.wy-menu-horiz li{display:inline-block}.wy-menu-horiz li:hover{background:rgba(255,255,255,0.1)}.wy-menu-horiz li.divide-left{border-left:solid 1px #404040}.wy-menu-horiz li.divide-right{border-right:solid 1px #404040}.wy-menu-horiz a{height:32px;display:inline-block;line-height:32px;padding:0 16px}.wy-menu-vertical{width:300px}.wy-menu-vertical header,.wy-menu-vertical p.caption{height:32px;display:inline-block;line-height:32px;padding:0 1.618em;margin-bottom:0;display:block;font-weight:bold;text-transform:uppercase;font-size:80%;color:#555;white-space:nowrap}.wy-menu-vertical ul{margin-bottom:0}.wy-menu-vertical li.divide-top{border-top:solid 1px #404040}.wy-menu-vertical li.divide-bottom{border-bottom:solid 1px #404040}.wy-menu-vertical li.current{background:#e3e3e3}.wy-menu-vertical li.current a{color:gray;border-right:solid 1px #c9c9c9;padding:0.4045em 2.427em}.wy-menu-vertical li.current a
:hover{background:#d6d6d6}.wy-menu-vertical li code,.wy-menu-vertical li .rst-content tt,.rst-content .wy-menu-vertical li tt{border:none;background:inherit;color:inherit;padding-left:0;padding-right:0}.wy-menu-vertical li span.toctree-expand{display:block;float:left;margin-left:-1.2em;font-size:0.8em;line-height:1.6em;color:#4d4d4d}.wy-menu-vertical li.on a,.wy-menu-vertical li.current>a{color:#404040;padding:0.4045em 1.618em;font-weight:bold;position:relative;background:#fcfcfc;border:none;border-bottom:solid 1px #c9c9c9;border-top:solid 1px #c9c9c9;padding-left:1.618em -4px}.wy-menu-vertical li.on a:hover,.wy-menu-vertical li.current>a:hover{background:#fcfcfc}.wy-menu-vertical li.on a:hover span.toctree-expand,.wy-menu-vertical li.current>a:hover span.toctree-expand{color:gray}.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li.current>a span.toctree-expand{display:block;font-size:0.8em;line-height:1.6em;color:#333}.wy-menu-vertical li.toctree-l1.current li.toctre
e-l2>ul,.wy-menu-vertical li.toctree-l2.current li.toctree-l3>ul{display:none}.wy-menu-vertical li.toctree-l1.current li.toctree-l2.current>ul,.wy-menu-vertical li.toctree-l2.current li.toctree-l3.current>ul{display:block}.wy-menu-vertical li.toctree-l2.current>a{background:#c9c9c9;padding:0.4045em 2.427em}.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a{display:block;background:#c9c9c9;padding:0.4045em 4.045em}.wy-menu-vertical li.toctree-l2 a:hover span.toctree-expand{color:gray}.wy-menu-vertical li.toctree-l2 span.toctree-expand{color:#a3a3a3}.wy-menu-vertical li.toctree-l3{font-size:0.9em}.wy-menu-vertical li.toctree-l3.current>a{background:#bdbdbd;padding:0.4045em 4.045em}.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a{display:block;background:#bdbdbd;padding:0.4045em 5.663em;border-top:none;border-bottom:none}.wy-menu-vertical li.toctree-l3 a:hover span.toctree-expand{color:gray}.wy-menu-vertical li.toctree-l3 span.toctree-expand{color:#969696}.wy-menu-vertical
li.toctree-l4{font-size:0.9em}.wy-menu-vertical li.current ul{display:block}.wy-menu-vertical li ul{margin-bottom:0;display:none}.wy-menu-vertical .local-toc li ul{display:block}.wy-menu-vertical li ul li a{margin-bottom:0;color:#b3b3b3;font-weight:normal}.wy-menu-vertical a{display:inline-block;line-height:18px;padding:0.4045em 1.618em;display:block;position:relative;font-size:90%;color:#b3b3b3}.wy-menu-vertical a:hover{background-color:#4e4a4a;cursor:pointer}.wy-menu-vertical a:hover span.toctree-expand{color:#b3b3b3}.wy-menu-vertical a:active{background-color:#2980B9;cursor:pointer;color:#fff}.wy-menu-vertical a:active span.toctree-expand{color:#fff}.wy-side-nav-search{display:block;width:300px;padding:0.809em;margin-bottom:0.809em;z-index:200;background-color:#2980B9;text-align:center;padding:0.809em;display:block;color:#fcfcfc;margin-bottom:0.809em}.wy-side-nav-search input[type=text]{width:100%;border-radius:50px;padding:6px 12px;border-color:#2472a4}.wy-side-nav-search img{di
splay:block;margin:auto auto 0.809em auto;height:45px;width:45px;background-color:#2980B9;padding:5px;border-radius:100%}.wy-side-nav-search>a,.wy-side-nav-search .wy-dropdown>a{color:#fcfcfc;font-size:100%;font-weight:bold;display:inline-block;padding:4px 6px;margin-bottom:0.809em}.wy-side-nav-search>a:hover,.wy-side-nav-search .wy-dropdown>a:hover{background:rgba(255,255,255,0.1)}.wy-side-nav-search>a img.logo,.wy-side-nav-search .wy-dropdown>a img.logo{display:block;margin:0 auto;height:auto;width:auto;border-radius:0;max-width:100%;background:transparent}.wy-side-nav-search>a.icon img.logo,.wy-side-nav-search .wy-dropdown>a.icon img.logo{margin-top:0.85em}.wy-side-nav-search>div.version{margin-top:-0.4045em;margin-bottom:0.809em;font-weight:normal;color:rgba(255,255,255,0.3)}.wy-nav .wy-menu-vertical header{color:#2980B9}.wy-nav .wy-menu-vertical a{color:#b3b3b3}.wy-nav .wy-menu-vertical a:hover{background-color:#2980B9;color:#fff}[data-menu-wrap]{-webkit-transition:all 0.2s eas
e-in;-moz-transition:all 0.2s ease-in;transition:all 0.2s ease-in;position:absolute;opacity:1;width:100%;opacity:0}[data-menu-wrap].move-center{left:0;right:auto;opacity:1}[data-menu-wrap].move-left{right:auto;left:-100%;opacity:0}[data-menu-wrap].move-right{right:-100%;left:auto;opacity:0}.wy-body-for-nav{background:left repeat-y #fcfcfc;background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAyRpVFh0WE1MOmNvbS5hZG9iZS54bXAAAAAAADw/eHBhY2tldCBiZWdpbj0i77u/IiBpZD0iVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkIj8+IDx4OnhtcG1ldGEgeG1sbnM6eD0iYWRvYmU6bnM6bWV0YS8iIHg6eG1wdGs9IkFkb2JlIFhNUCBDb3JlIDUuMy1jMDExIDY2LjE0NTY2MSwgMjAxMi8wMi8wNi0xNDo1NjoyNyAgICAgICAgIj4gPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4gPHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9IiIgeG1sbnM6eG1wPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIiB4bWxuczp4bXBNTT0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wL21tLyIgeG1sbnM6c3RSZWY9I
mh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9zVHlwZS9SZXNvdXJjZVJlZiMiIHhtcDpDcmVhdG9yVG9vbD0iQWRvYmUgUGhvdG9zaG9wIENTNiAoTWFjaW50b3NoKSIgeG1wTU06SW5zdGFuY2VJRD0ieG1wLmlpZDoxOERBMTRGRDBFMUUxMUUzODUwMkJCOThDMEVFNURFMCIgeG1wTU06RG9jdW1lbnRJRD0ieG1wLmRpZDoxOERBMTRGRTBFMUUxMUUzODUwMkJCOThDMEVFNURFMCI+IDx4bXBNTTpEZXJpdmVkRnJvbSBzdFJlZjppbnN0YW5jZUlEPSJ4bXAuaWlkOjE4REExNEZCMEUxRTExRTM4NTAyQkI5OEMwRUU1REUwIiBzdFJlZjpkb2N1bWVudElEPSJ4bXAuZGlkOjE4REExNEZDMEUxRTExRTM4NTAyQkI5OEMwRUU1REUwIi8+IDwvcmRmOkRlc2NyaXB0aW9uPiA8L3JkZjpSREY+IDwveDp4bXBtZXRhPiA8P3hwYWNrZXQgZW5kPSJyIj8+EwrlwAAAAA5JREFUeNpiMDU0BAgwAAE2AJgB9BnaAAAAAElFTkSuQmCC);background-size:300px 1px}.wy-grid-for-nav{position:absolute;width:100%;height:100%}.wy-nav-side{position:fixed;top:0;bottom:0;left:0;padding-bottom:2em;width:300px;overflow-x:hidden;overflow-y:hidden;min-height:100%;background:#343131;z-index:200}.wy-side-scroll{width:320px;position:relative;overflow-x:hidden;overflow-y:scroll;height:100%}.wy-nav-top{display:none;backgroun
d:#2980B9;color:#fff;padding:0.4045em 0.809em;position:relative;line-height:50px;text-align:center;font-size:100%;*zoom:1}.wy-nav-top:before,.wy-nav-top:after{display:table;content:""}.wy-nav-top:after{clear:both}.wy-nav-top a{color:#fff;font-weight:bold}.wy-nav-top img{margin-right:12px;height:45px;width:45px;background-color:#2980B9;padding:5px;border-radius:100%}.wy-nav-top i{font-size:30px;float:left;cursor:pointer}.wy-nav-content-wrap{margin-left:300px;background:#fcfcfc;min-height:100%}.wy-nav-content{padding:1.618em 3.236em;height:100%;max-width:800px;margin:auto}.wy-body-mask{position:fixed;width:100%;height:100%;background:rgba(0,0,0,0.2);display:none;z-index:499}.wy-body-mask.on{display:block}footer{color:#999}footer p{margin-bottom:12px}footer span.commit code,footer span.commit .rst-content tt,.rst-content footer span.commit tt{padding:0px;font-family:Consolas,"Andale Mono WT","Andale Mono","Lucida Console","Lucida Sans Typewriter","DejaVu Sans Mo
<TRUNCATED>
[06/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/code.html
----------------------------------------------------------------------
diff --git a/code.html b/code.html
new file mode 100644
index 0000000..4bf307d
--- /dev/null
+++ b/code.html
@@ -0,0 +1,3517 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>API Reference — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="prev" title="FAQ" href="faq.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">API Reference</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#operators">Operators</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#baseoperator">BaseOperator</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#basesensoroperator">BaseSensorOperator</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#module-airflow.operators">Operator API</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#community-contributed-operators">Community-contributed Operators</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="#macros">Macros</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#default-variables">Default Variables</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#id2">Macros</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="#models">Models</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#module-airflow.hooks">Hooks</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#community-contributed-hooks">Community contributed hooks</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="#executors">Executors</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#community-contributed-executors">Community-contributed executors</a></li>
+</ul>
+</li>
+</ul>
+</li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>API Reference</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/code.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="api-reference">
+<h1>API Reference<a class="headerlink" href="#api-reference" title="Permalink to this headline">�</a></h1>
+<div class="section" id="operators">
+<h2>Operators<a class="headerlink" href="#operators" title="Permalink to this headline">�</a></h2>
+<p>Operators allow for generation of certain types of tasks that become nodes in
+the DAG when instantiated. All operators derive from BaseOperator and
+inherit many attributes and methods that way. Refer to the BaseOperator
+documentation for more details.</p>
+<p>There are 3 main types of operators:</p>
+<ul class="simple">
+<li>Operators that performs an <strong>action</strong>, or tell another system to
+perform an action</li>
+<li><strong>Transfer</strong> operators move data from one system to another</li>
+<li><strong>Sensors</strong> are a certain type of operator that will keep running until a
+certain criterion is met. Examples include a specific file landing in HDFS or
+S3, a partition appearing in Hive, or a specific time of the day. Sensors
+are derived from <code class="docutils literal"><span class="pre">BaseSensorOperator</span></code> and run a poke
+method at a specified <code class="docutils literal"><span class="pre">poke_interval</span></code> until it returns <code class="docutils literal"><span class="pre">True</span></code>.</li>
+</ul>
+<div class="section" id="baseoperator">
+<h3>BaseOperator<a class="headerlink" href="#baseoperator" title="Permalink to this headline">�</a></h3>
+<p>All operators are derived from <code class="docutils literal"><span class="pre">BaseOperator</span></code> and acquire much
+functionality through inheritance. Since this is the core of the engine,
+it’s worth taking the time to understand the parameters of <code class="docutils literal"><span class="pre">BaseOperator</span></code>
+to understand the primitive features that can be leveraged in your
+DAGs.</p>
+<dl class="class">
+<dt id="airflow.models.BaseOperator">
+<em class="property">class </em><code class="descclassname">airflow.models.</code><code class="descname">BaseOperator</code><span class="sig-paren">(</span><em>task_id</em>, <em>owner='airflow'</em>, <em>email=None</em>, <em>email_on_retry=True</em>, <em>email_on_failure=True</em>, <em>retries=0</em>, <em>retry_delay=datetime.timedelta(0</em>, <em>300)</em>, <em>start_date=None</em>, <em>end_date=None</em>, <em>schedule_interval=None</em>, <em>depends_on_past=False</em>, <em>wait_for_downstream=False</em>, <em>dag=None</em>, <em>params=None</em>, <em>default_args=None</em>, <em>adhoc=False</em>, <em>priority_weight=1</em>, <em>queue='default'</em>, <em>pool=None</em>, <em>sla=None</em>, <em>execution_timeout=None</em>, <em>on_failure_callback=None</em>, <em>on_success_callback=None</em>, <em>on_retry_callback=None</em>, <em>trigger_rule=u'all_success'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/models.ht
ml#BaseOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.models.BaseOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Abstract base class for all operators. Since operators create objects that
+become node in the dag, BaseOperator contains many recursive methods for
+dag crawling behavior. To derive this class, you are expected to override
+the constructor as well as the ‘execute’ method.</p>
+<p>Operators derived from this task should perform or trigger certain tasks
+synchronously (wait for completion). Example of operators could be an
+operator the runs a Pig job (PigOperator), a sensor operator that
+waits for a partition to land in Hive (HiveSensorOperator), or one that
+moves data from Hive to MySQL (Hive2MySqlOperator). Instances of these
+operators (tasks) target specific operations, running specific scripts,
+functions or data transfers.</p>
+<p>This class is abstract and shouldn’t be instantiated. Instantiating a
+class derived from this one results in the creation of a task object,
+which ultimately becomes a node in DAG objects. Task dependencies should
+be set by using the set_upstream and/or set_downstream methods.</p>
+<p>Note that this class is derived from SQLAlchemy’s Base class, which
+allows us to push metadata regarding tasks to the database. Deriving this
+classes needs to implement the polymorphic specificities documented in
+SQLAlchemy. This should become clear while reading the code for other
+operators.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>task_id</strong> (<em>string</em>) – a unique, meaningful id for the task</li>
+<li><strong>owner</strong> (<em>string</em>) – the owner of the task, using the unix username is recommended</li>
+<li><strong>retries</strong> (<em>int</em>) – the number of retries that should be performed before
+failing the task</li>
+<li><strong>retry_delay</strong> (<em>timedelta</em>) – delay between retries</li>
+<li><strong>start_date</strong> (<em>datetime</em>) – The <code class="docutils literal"><span class="pre">start_date</span></code> for the task, determines
+the <code class="docutils literal"><span class="pre">execution_date</span></code> for the first task instance. The best practice
+is to have the start_date rounded
+to your DAG’s <code class="docutils literal"><span class="pre">schedule_interval</span></code>. Daily jobs have their start_date
+some day at 00:00:00, hourly jobs have their start_date at 00:00
+of a specific hour. Note that Airflow simply looks at the latest
+<code class="docutils literal"><span class="pre">execution_date</span></code> and adds the <code class="docutils literal"><span class="pre">schedule_interval</span></code> to determine
+the next <code class="docutils literal"><span class="pre">execution_date</span></code>. It is also very important
+to note that different tasks’ dependencies
+need to line up in time. If task A depends on task B and their
+start_date are offset in a way that their execution_date don’t line
+up, A’s dependencies will never be met. If you are looking to delay
+a task, for example running a daily task at 2AM, look into the
+<code class="docutils literal"><span class="pre">TimeSensor</span></code> and <code class="docutils literal"><span class="pre">TimeDeltaSensor</span></code>. We advise against using
+dynamic <code class="docutils literal"><span class="pre">start_date</span></code> and recommend using fixed ones. Read the
+FAQ entry about start_date for more information.</li>
+<li><strong>end_date</strong> (<em>datetime</em>) – if specified, the scheduler won’t go beyond this date</li>
+<li><strong>depends_on_past</strong> (<em>bool</em>) – when set to true, task instances will run
+sequentially while relying on the previous task’s schedule to
+succeed. The task instance for the start_date is allowed to run.</li>
+<li><strong>wait_for_downstream</strong> (<em>bool</em>) – when set to true, an instance of task
+X will wait for tasks immediately downstream of the previous instance
+of task X to finish successfully before it runs. This is useful if the
+different instances of a task X alter the same asset, and this asset
+is used by tasks downstream of task X. Note that depends_on_past
+is forced to True wherever wait_for_downstream is used.</li>
+<li><strong>queue</strong> (<em>str</em>) – which queue to target when running this job. Not
+all executors implement queue management, the CeleryExecutor
+does support targeting specific queues.</li>
+<li><strong>dag</strong> (<a class="reference internal" href="#airflow.models.DAG" title="airflow.models.DAG"><em>DAG</em></a>) – a reference to the dag the task is attached to (if any)</li>
+<li><strong>priority_weight</strong> (<em>int</em>) – priority weight of this task against other task.
+This allows the executor to trigger higher priority tasks before
+others when things get backed up.</li>
+<li><strong>pool</strong> (<em>str</em>) – the slot pool this task should run in, slot pools are a
+way to limit concurrency for certain tasks</li>
+<li><strong>sla</strong> (<em>datetime.timedelta</em>) – time by which the job is expected to succeed. Note that
+this represents the <code class="docutils literal"><span class="pre">timedelta</span></code> after the period is closed. For
+example if you set an SLA of 1 hour, the scheduler would send dan email
+soon after 1:00AM on the <code class="docutils literal"><span class="pre">2016-01-02</span></code> if the <code class="docutils literal"><span class="pre">2016-01-01</span></code> instance
+has not succeede yet.
+The scheduler pays special attention for jobs with an SLA and
+sends alert
+emails for sla misses. SLA misses are also recorded in the database
+for future reference. All tasks that share the same SLA time
+get bundled in a single email, sent soon after that time. SLA
+notification are sent once and only once for each task instance.</li>
+<li><strong>execution_timeout</strong> (<em>datetime.timedelta</em>) – max time allowed for the execution of
+this task instance, if it goes beyond it will raise and fail.</li>
+<li><strong>on_failure_callback</strong> (<em>callable</em>) – a function to be called when a task instance
+of this task fails. a context dictionary is passed as a single
+parameter to this function. Context contains references to related
+objects to the task instance and is documented under the macros
+section of the API.</li>
+<li><strong>on_retry_callback</strong> – much like the <code class="docutils literal"><span class="pre">on_failure_callback</span></code> excepts
+that it is executed when retries occur.</li>
+<li><strong>on_success_callback</strong> (<em>callable</em>) – much like the <code class="docutils literal"><span class="pre">on_failure_callback</span></code> excepts
+that it is executed when the task succeeds.</li>
+<li><strong>trigger_rule</strong> (<em>str</em>) – defines the rule by which dependencies are applied
+for the task to get triggered. Options are:
+<code class="docutils literal"><span class="pre">{</span> <span class="pre">all_success</span> <span class="pre">|</span> <span class="pre">all_failed</span> <span class="pre">|</span> <span class="pre">all_done</span> <span class="pre">|</span> <span class="pre">one_success</span> <span class="pre">|</span>
+<span class="pre">one_failed</span> <span class="pre">|</span> <span class="pre">dummy}</span></code>
+default is <code class="docutils literal"><span class="pre">all_success</span></code>. Options can be set as string or
+using the constants defined in the static class
+<code class="docutils literal"><span class="pre">airflow.utils.TriggerRule</span></code></li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</div>
+<div class="section" id="basesensoroperator">
+<h3>BaseSensorOperator<a class="headerlink" href="#basesensoroperator" title="Permalink to this headline">�</a></h3>
+<p>All sensors are derived from <code class="docutils literal"><span class="pre">BaseSensorOperator</span></code>. All sensors inherit
+the <code class="docutils literal"><span class="pre">timeout</span></code> and <code class="docutils literal"><span class="pre">poke_interval</span></code> on top of the <code class="docutils literal"><span class="pre">BaseOperator</span></code>
+attributes.</p>
+<dl class="class">
+<dt id="airflow.operators.sensors.BaseSensorOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.sensors.</code><code class="descname">BaseSensorOperator</code><span class="sig-paren">(</span><em>poke_interval=60</em>, <em>timeout=604800</em>, <em>soft_fail=False</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/sensors.html#BaseSensorOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.sensors.BaseSensorOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Sensor operators are derived from this class an inherit these attributes.</p>
+<dl class="docutils">
+<dt>Sensor operators keep executing at a time interval and succeed when</dt>
+<dd>a criteria is met and fail if and when they time out.</dd>
+</dl>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>soft_fail</strong> (<em>bool</em>) – Set to true to mark the task as SKIPPED on failure</li>
+<li><strong>poke_interval</strong> (<em>int</em>) – Time in seconds that the job should wait in
+between each tries</li>
+<li><strong>timeout</strong> (<em>int</em>) – Time, in seconds before the task times out and fails.</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</div>
+<div class="section" id="module-airflow.operators">
+<span id="operator-api"></span><h3>Operator API<a class="headerlink" href="#module-airflow.operators" title="Permalink to this headline">�</a></h3>
+<dl class="class">
+<dt id="airflow.operators.BashOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">BashOperator</code><span class="sig-paren">(</span><em>bash_command</em>, <em>xcom_push=False</em>, <em>env=None</em>, <em>output_encoding='utf-8'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/bash_operator.html#BashOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.BashOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Execute a Bash script, command or set of commands.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>bash_command</strong> (<em>string</em>) – The command, set of commands or reference to a
+bash script (must be ‘.sh’) to be executed.</li>
+<li><strong>env</strong> (<em>dict</em>) – If env is not None, it must be a mapping that defines the
+environment variables for the new process; these are used instead
+of inheriting the current process environment, which is the default
+behavior. (templated)</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+<dl class="method">
+<dt id="airflow.operators.BashOperator.execute">
+<code class="descname">execute</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/bash_operator.html#BashOperator.execute"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.BashOperator.execute" title="Permalink to this definition">�</a></dt>
+<dd><p>Execute the bash command in a temporary directory
+which will be cleaned afterwards</p>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.BranchPythonOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">BranchPythonOperator</code><span class="sig-paren">(</span><em>python_callable</em>, <em>op_args=None</em>, <em>op_kwargs=None</em>, <em>provide_context=False</em>, <em>templates_dict=None</em>, <em>templates_exts=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/python_operator.html#BranchPythonOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.BranchPythonOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">python_operator.PythonOperator</span></code></p>
+<p>Allows a workflow to “branch” or follow a single path following the
+execution of this task.</p>
+<p>It derives the PythonOperator and expects a Python function that returns
+the task_id to follow. The task_id returned should point to a task
+directly downstream from {self}. All other “branches” or
+directly downstream tasks are marked with a state of <code class="docutils literal"><span class="pre">skipped</span></code> so that
+these paths can’t move forward. The <code class="docutils literal"><span class="pre">skipped</span></code> states are propageted
+downstream to allow for the DAG state to fill up and the DAG run’s state
+to be inferred.</p>
+<p>Note that using tasks with <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> downstream from
+<code class="docutils literal"><span class="pre">BranchPythonOperator</span></code> is logically unsound as <code class="docutils literal"><span class="pre">skipped</span></code> status
+will invariably lead to block tasks that depend on their past successes.
+<code class="docutils literal"><span class="pre">skipped</span></code> states propagates where all directly upstream tasks are
+<code class="docutils literal"><span class="pre">skipped</span></code>.</p>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.TriggerDagRunOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">TriggerDagRunOperator</code><span class="sig-paren">(</span><em>trigger_dag_id</em>, <em>python_callable</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/dagrun_operator.html#TriggerDagRunOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.TriggerDagRunOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Triggers a DAG run for a specified <code class="docutils literal"><span class="pre">dag_id</span></code> if a criteria is met</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>trigger_dag_id</strong> (<em>str</em>) – the dag_id to trigger</li>
+<li><strong>python_callable</strong> (<em>python callable</em>) – a reference to a python function that will be
+called while passing it the <code class="docutils literal"><span class="pre">context</span></code> object and a placeholder
+object <code class="docutils literal"><span class="pre">obj</span></code> for your callable to fill and return if you want
+a DagRun created. This <code class="docutils literal"><span class="pre">obj</span></code> object contains a <code class="docutils literal"><span class="pre">run_id</span></code> and
+<code class="docutils literal"><span class="pre">payload</span></code> attribute that you can modify in your function.
+The <code class="docutils literal"><span class="pre">run_id</span></code> should be a unique identifier for that DAG run, and
+the payload has to be a picklable object that will be made available
+to your tasks while executing that DAG run. Your function header
+should look like <code class="docutils literal"><span class="pre">def</span> <span class="pre">foo(context,</span> <span class="pre">dag_run_obj):</span></code></li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.DummyOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">DummyOperator</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/dummy_operator.html#DummyOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.DummyOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Operator that does literally nothing. It can be used to group tasks in a
+DAG.</p>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.EmailOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">EmailOperator</code><span class="sig-paren">(</span><em>to</em>, <em>subject</em>, <em>html_content</em>, <em>files=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/email_operator.html#EmailOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.EmailOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Sends an email.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>to</strong> (<em>list or string (comma or semicolon delimited)</em>) – list of emails to send the email to</li>
+<li><strong>subject</strong> (<em>string</em>) – subject line for the email (templated)</li>
+<li><strong>html_content</strong> (<em>string</em>) – content of the email (templated), html markup
+is allowed</li>
+<li><strong>files</strong> (<em>list</em>) – file names to attach in email</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.ExternalTaskSensor">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">ExternalTaskSensor</code><span class="sig-paren">(</span><em>external_dag_id</em>, <em>external_task_id</em>, <em>allowed_states=None</em>, <em>execution_delta=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#ExternalTaskSensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.ExternalTaskSensor" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.operators.sensors.BaseSensorOperator" title="airflow.operators.sensors.BaseSensorOperator"><code class="xref py py-class docutils literal"><span class="pre">sensors.BaseSensorOperator</span></code></a></p>
+<p>Waits for a task to complete in a different DAG</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>external_dag_id</strong> (<em>string</em>) – The dag_id that contains the task you want to
+wait for</li>
+<li><strong>external_task_id</strong> (<em>string</em>) – The task_id that contains the task you want to
+wait for</li>
+<li><strong>allowed_states</strong> (<em>list</em>) – list of allowed states, default is <code class="docutils literal"><span class="pre">['success']</span></code></li>
+<li><strong>execution_delta</strong> (<em>datetime.timedelta</em>) – time difference with the previous execution to
+look at, the default is the same execution_date as the current task.
+For yesterday, use [positive!] datetime.timedelta(days=1)</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.GenericTransfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">GenericTransfer</code><span class="sig-paren">(</span><em>sql</em>, <em>destination_table</em>, <em>source_conn_id</em>, <em>destination_conn_id</em>, <em>preoperator=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/generic_transfer.html#GenericTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.GenericTransfer" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Moves data from a connection to another, assuming that they both
+provide the required methods in their respective hooks. The source hook
+needs to expose a <cite>get_records</cite> method, and the destination a
+<cite>insert_rows</cite> method.</p>
+<p>This is mean to be used on small-ish datasets that fit in memory.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>sql</strong> (<em>str</em>) – SQL query to execute against the source database</li>
+<li><strong>destination_table</strong> (<em>str</em>) – target table</li>
+<li><strong>source_conn_id</strong> (<em>str</em>) – source connection</li>
+<li><strong>destination_conn_id</strong> (<em>str</em>) – source connection</li>
+<li><strong>preoperator</strong> (<em>str or list of str</em>) – sql statement or list of statements to be
+executed prior to loading the data</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.HdfsSensor">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">HdfsSensor</code><span class="sig-paren">(</span><em>filepath</em>, <em>hdfs_conn_id='hdfs_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#HdfsSensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.HdfsSensor" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.operators.sensors.BaseSensorOperator" title="airflow.operators.sensors.BaseSensorOperator"><code class="xref py py-class docutils literal"><span class="pre">sensors.BaseSensorOperator</span></code></a></p>
+<p>Waits for a file or folder to land in HDFS</p>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.Hive2SambaOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">Hive2SambaOperator</code><span class="sig-paren">(</span><em>hql</em>, <em>destination_filepath</em>, <em>samba_conn_id='samba_default'</em>, <em>hiveserver2_conn_id='hiveserver2_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hive_to_samba_operator.html#Hive2SambaOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.Hive2SambaOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Executes hql code in a specific Hive database and loads the
+results of the query as a csv to a Samba location.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>hql</strong> (<em>string</em>) – the hql to be exported</li>
+<li><strong>hiveserver2_conn_id</strong> (<em>string</em>) – reference to the hiveserver2 service</li>
+<li><strong>samba_conn_id</strong> (<em>string</em>) – reference to the samba destination</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.HiveOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">HiveOperator</code><span class="sig-paren">(</span><em>hql</em>, <em>hive_cli_conn_id='hive_cli_default'</em>, <em>schema='default'</em>, <em>hiveconf_jinja_translate=False</em>, <em>script_begin_tag=None</em>, <em>run_as_owner=False</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hive_operator.html#HiveOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.HiveOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Executes hql code in a specific Hive database.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>hql</strong> (<em>string</em>) – the hql to be executed</li>
+<li><strong>hive_cli_conn_id</strong> (<em>string</em>) – reference to the Hive database</li>
+<li><strong>hiveconf_jinja_translate</strong> (<em>boolean</em>) – when True, hiveconf-type templating
+${var} gets translated into jinja-type templating {{ var }}. Note that
+you may want to use this along with the
+<code class="docutils literal"><span class="pre">DAG(user_defined_macros=myargs)</span></code> parameter. View the DAG
+object documentation for more details.</li>
+<li><strong>script_begin_tag</strong> (<em>str</em>) – If defined, the operator will get rid of the
+part of the script before the first occurrence of <cite>script_begin_tag</cite></li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.HivePartitionSensor">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">HivePartitionSensor</code><span class="sig-paren">(</span><em>table</em>, <em>partition="ds='{{ ds }}'"</em>, <em>metastore_conn_id='metastore_default'</em>, <em>schema='default'</em>, <em>poke_interval=180</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#HivePartitionSensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.HivePartitionSensor" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.operators.sensors.BaseSensorOperator" title="airflow.operators.sensors.BaseSensorOperator"><code class="xref py py-class docutils literal"><span class="pre">sensors.BaseSensorOperator</span></code></a></p>
+<p>Waits for a partition to show up in Hive</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>table</strong> (<em>string</em>) – The name of the table to wait for, supports the dot
+notation (my_database.my_table)</li>
+<li><strong>partition</strong> (<em>string</em>) – The partition clause to wait for. This is passed as
+is to the Metastore Thrift client “get_partitions_by_filter” method,
+and apparently supports SQL like notation as in <cite>ds=‘2015-01-01’
+AND type=’value’</cite> and > < sings as in “ds>=2015-01-01”</li>
+<li><strong>metastore_conn_id</strong> (<em>str</em>) – reference to the metastore thrift service
+connection id</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.HiveToDruidTransfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">HiveToDruidTransfer</code><span class="sig-paren">(</span><em>sql</em>, <em>druid_datasource</em>, <em>ts_dim</em>, <em>metric_spec=None</em>, <em>hive_cli_conn_id='hive_cli_default'</em>, <em>druid_ingest_conn_id='druid_ingest_default'</em>, <em>metastore_conn_id='metastore_default'</em>, <em>hadoop_dependency_coordinates=None</em>, <em>intervals=None</em>, <em>num_shards=-1</em>, <em>target_partition_size=-1</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hive_to_druid.html#HiveToDruidTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.HiveToDruidTransfer" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Moves data from Hive to Druid, [del]note that for now the data is loaded
+into memory before being pushed to Druid, so this operator should
+be used for smallish amount of data.[/del]</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>sql</strong> (<em>str</em>) – SQL query to execute against the Druid database</li>
+<li><strong>druid_datasource</strong> (<em>str</em>) – the datasource you want to ingest into in druid</li>
+<li><strong>ts_dim</strong> (<em>str</em>) – the timestamp dimension</li>
+<li><strong>metric_spec</strong> (<em>list</em>) – the metrics you want to define for your data</li>
+<li><strong>hive_cli_conn_id</strong> (<em>str</em>) – the hive connection id</li>
+<li><strong>druid_ingest_conn_id</strong> (<em>str</em>) – the druid ingest connection id</li>
+<li><strong>metastore_conn_id</strong> (<em>str</em>) – the metastore connection id</li>
+<li><strong>hadoop_dependency_coordinates</strong> (<em>list of str</em>) – list of coordinates to squeeze
+int the ingest json</li>
+<li><strong>intervals</strong> (<em>list</em>) – list of time intervals that defines segments, this
+is passed as is to the json object</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.HiveToMySqlTransfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">HiveToMySqlTransfer</code><span class="sig-paren">(</span><em>sql</em>, <em>mysql_table</em>, <em>hiveserver2_conn_id='hiveserver2_default'</em>, <em>mysql_conn_id='mysql_default'</em>, <em>mysql_preoperator=None</em>, <em>mysql_postoperator=None</em>, <em>bulk_load=False</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/hive_to_mysql.html#HiveToMySqlTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.HiveToMySqlTransfer" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Moves data from Hive to MySQL, note that for now the data is loaded
+into memory before being pushed to MySQL, so this operator should
+be used for smallish amount of data.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>sql</strong> (<em>str</em>) – SQL query to execute against the MySQL database</li>
+<li><strong>mysql_table</strong> (<em>str</em>) – target MySQL table, use dot notation to target a
+specific database</li>
+<li><strong>mysql_conn_id</strong> (<em>str</em>) – source mysql connection</li>
+<li><strong>hiveserver2_conn_id</strong> (<em>str</em>) – destination hive connection</li>
+<li><strong>mysql_preoperator</strong> (<em>str</em>) – sql statement to run against mysql prior to
+import, typically use to truncate of delete in place of the data
+coming in, allowing the task to be idempotent (running the task
+twice won’t double load data)</li>
+<li><strong>mysql_postoperator</strong> (<em>str</em>) – sql statement to run against mysql after the
+import, typically used to move data from staging to production
+and issue cleanup commands.</li>
+<li><strong>bulk_load</strong> (<em>bool</em>) – flag to use bulk_load option. This loads mysql directly
+from a tab-delimited text file using the LOAD DATA LOCAL INFILE command.
+This option requires an extra connection parameter for the
+destination MySQL connection: {‘local_infile’: true}.</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.SimpleHttpOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">SimpleHttpOperator</code><span class="sig-paren">(</span><em>endpoint</em>, <em>method='POST'</em>, <em>data=None</em>, <em>headers=None</em>, <em>response_check=None</em>, <em>extra_options=None</em>, <em>http_conn_id='http_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/http_operator.html#SimpleHttpOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.SimpleHttpOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Calls an endpoint on an HTTP system to execute an action</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>http_conn_id</strong> (<em>string</em>) – The connection to run the sensor against</li>
+<li><strong>endpoint</strong> (<em>string</em>) – The relative part of the full url</li>
+<li><strong>method</strong> (<em>string</em>) – The HTTP method to use, default = “POST”</li>
+<li><strong>data</strong> (<em>For POST/PUT, depends on the content-type parameter,
+for GET a dictionary of key/value string pairs</em>) – The data to pass. POST-data in POST/PUT and params
+in the URL for a GET request.</li>
+<li><strong>headers</strong> (<em>a dictionary of string key/value pairs</em>) – The HTTP headers to be added to the GET request</li>
+<li><strong>response_check</strong> (<em>A lambda or defined function.</em>) – A check against the ‘requests’ response object.
+Returns True for ‘pass’ and False otherwise.</li>
+<li><strong>extra_options</strong> (<em>A dictionary of options, where key is string and value
+depends on the option that's being modified.</em>) – Extra options for the ‘requests’ library, see the
+‘requests’ documentation (options to modify timeout, ssl, etc.)</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.HttpSensor">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">HttpSensor</code><span class="sig-paren">(</span><em>endpoint</em>, <em>http_conn_id='http_default'</em>, <em>params=None</em>, <em>headers=None</em>, <em>response_check=None</em>, <em>extra_options=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#HttpSensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.HttpSensor" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.operators.sensors.BaseSensorOperator" title="airflow.operators.sensors.BaseSensorOperator"><code class="xref py py-class docutils literal"><span class="pre">sensors.BaseSensorOperator</span></code></a></p>
+<dl class="docutils">
+<dt>Executes a HTTP get statement and returns False on failure:</dt>
+<dd>404 not found or response_check function returned False</dd>
+</dl>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>http_conn_id</strong> (<em>string</em>) – The connection to run the sensor against</li>
+<li><strong>endpoint</strong> (<em>string</em>) – The relative part of the full url</li>
+<li><strong>params</strong> (<em>a dictionary of string key/value pairs</em>) – The parameters to be added to the GET url</li>
+<li><strong>headers</strong> (<em>a dictionary of string key/value pairs</em>) – The HTTP headers to be added to the GET request</li>
+<li><strong>response_check</strong> (<em>A lambda or defined function.</em>) – A check against the ‘requests’ response object.
+Returns True for ‘pass’ and False otherwise.</li>
+<li><strong>extra_options</strong> (<em>A dictionary of options, where key is string and value
+depends on the option that's being modified.</em>) – Extra options for the ‘requests’ library, see the
+‘requests’ documentation (options to modify timeout, ssl, etc.)</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.MetastorePartitionSensor">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">MetastorePartitionSensor</code><span class="sig-paren">(</span><em>table</em>, <em>partition_name</em>, <em>schema='default'</em>, <em>mysql_conn_id='metastore_mysql'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#MetastorePartitionSensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.MetastorePartitionSensor" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">sensors.SqlSensor</span></code></p>
+<p>An alternative to the HivePartitionSensor that talk directly to the
+MySQL db. This was created as a result of observing sub optimal
+queries generated by the Metastore thrift service when hitting
+subpartitioned tables. The Thrift service’s queries were written in a
+way that wouldn’t leverage the indexes.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>schema</strong> (<em>str</em>) – the schema</li>
+<li><strong>table</strong> (<em>str</em>) – the table</li>
+<li><strong>partition_name</strong> (<em>str</em>) – the partition name, as defined in the PARTITIONS
+table of the Metastore. Order of the fields does matter.
+Examples: <code class="docutils literal"><span class="pre">ds=2016-01-01</span></code> or
+<code class="docutils literal"><span class="pre">ds=2016-01-01/sub=foo</span></code> for a sub partitioned table</li>
+<li><strong>mysql_conn_id</strong> (<em>str</em>) – a reference to the MySQL conn_id for the metastore</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.MsSqlOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">MsSqlOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>mssql_conn_id='mssql_default'</em>, <em>parameters=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/mssql_operator.html#MsSqlOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.MsSqlOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Executes sql code in a specific Microsoft SQL database
+:param mssql_conn_id: reference to a specific mssql database
+:type mssql_conn_id: string
+:param sql: the sql code to be executed
+:type sql: string or string pointing to a template file.
+File must have a ‘.sql’ extensions.</p>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.MsSqlToHiveTransfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">MsSqlToHiveTransfer</code><span class="sig-paren">(</span><em>sql</em>, <em>hive_table</em>, <em>create=True</em>, <em>recreate=False</em>, <em>partition=None</em>, <em>delimiter=u'x01'</em>, <em>mssql_conn_id='mssql_default'</em>, <em>hive_cli_conn_id='hive_cli_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/mssql_to_hive.html#MsSqlToHiveTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.MsSqlToHiveTransfer" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Moves data from Microsoft SQL Server to Hive. The operator runs
+your query against Microsoft SQL Server, stores the file locally
+before loading it into a Hive table. If the <code class="docutils literal"><span class="pre">create</span></code> or
+<code class="docutils literal"><span class="pre">recreate</span></code> arguments are set to <code class="docutils literal"><span class="pre">True</span></code>,
+a <code class="docutils literal"><span class="pre">CREATE</span> <span class="pre">TABLE</span></code> and <code class="docutils literal"><span class="pre">DROP</span> <span class="pre">TABLE</span></code> statements are generated.
+Hive data types are inferred from the cursor’s metadata.
+Note that the table generated in Hive uses <code class="docutils literal"><span class="pre">STORED</span> <span class="pre">AS</span> <span class="pre">textfile</span></code>
+which isn’t the most efficient serialization format. If a
+large amount of data is loaded and/or if the table gets
+queried considerably, you may want to use this operator only to
+stage the data into a temporary table before loading it into its
+final destination using a <code class="docutils literal"><span class="pre">HiveOperator</span></code>.
+:param sql: SQL query to execute against the Microsoft SQL Server database
+:type sql: str
+:param hive_table: target Hive table, use dot notation to target a
+specific database
+:type hive_table: str
+:param create: whether to create the table if it doesn’t exist
+:type create: bool
+:param recreate: whether to drop and recreate the table at every execution
+:type recreate: bool
+:param partition: target partition as a dict of partition columns and values
+:type partition: dict
+:param delimiter: field delimiter in the file
+:type delimiter: str
+:param mssql_conn_id: source Microsoft SQL Server connection
+:type mssql_conn_id: str
+:param hive_conn_id: destination hive connection
+:type hive_conn_id: str</p>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.MySqlOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">MySqlOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>mysql_conn_id='mysql_default'</em>, <em>parameters=None</em>, <em>autocommit=False</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/mysql_operator.html#MySqlOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.MySqlOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Executes sql code in a specific MySQL database</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>mysql_conn_id</strong> (<em>string</em>) – reference to a specific mysql database</li>
+<li><strong>sql</strong> (<em>Can receive a str representing a sql statement,
+a list of str (sql statements), or reference to a template file.
+Template reference are recognized by str ending in '.sql'</em>) – the sql code to be executed</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.MySqlToHiveTransfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">MySqlToHiveTransfer</code><span class="sig-paren">(</span><em>sql</em>, <em>hive_table</em>, <em>create=True</em>, <em>recreate=False</em>, <em>partition=None</em>, <em>delimiter=u'x01'</em>, <em>mysql_conn_id='mysql_default'</em>, <em>hive_cli_conn_id='hive_cli_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/mysql_to_hive.html#MySqlToHiveTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.MySqlToHiveTransfer" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Moves data from MySql to Hive. The operator runs your query against
+MySQL, stores the file locally before loading it into a Hive table.
+If the <code class="docutils literal"><span class="pre">create</span></code> or <code class="docutils literal"><span class="pre">recreate</span></code> arguments are set to <code class="docutils literal"><span class="pre">True</span></code>,
+a <code class="docutils literal"><span class="pre">CREATE</span> <span class="pre">TABLE</span></code> and <code class="docutils literal"><span class="pre">DROP</span> <span class="pre">TABLE</span></code> statements are generated.
+Hive data types are inferred from the cursor’s metadata. Note that the
+table generated in Hive uses <code class="docutils literal"><span class="pre">STORED</span> <span class="pre">AS</span> <span class="pre">textfile</span></code>
+which isn’t the most efficient serialization format. If a
+large amount of data is loaded and/or if the table gets
+queried considerably, you may want to use this operator only to
+stage the data into a temporary table before loading it into its
+final destination using a <code class="docutils literal"><span class="pre">HiveOperator</span></code>.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>sql</strong> (<em>str</em>) – SQL query to execute against the MySQL database</li>
+<li><strong>hive_table</strong> (<em>str</em>) – target Hive table, use dot notation to target a
+specific database</li>
+<li><strong>create</strong> (<em>bool</em>) – whether to create the table if it doesn’t exist</li>
+<li><strong>recreate</strong> (<em>bool</em>) – whether to drop and recreate the table at every
+execution</li>
+<li><strong>partition</strong> (<em>dict</em>) – target partition as a dict of partition columns
+and values</li>
+<li><strong>delimiter</strong> (<em>str</em>) – field delimiter in the file</li>
+<li><strong>mysql_conn_id</strong> (<em>str</em>) – source mysql connection</li>
+<li><strong>hive_conn_id</strong> (<em>str</em>) – destination hive connection</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.PostgresOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">PostgresOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>postgres_conn_id='postgres_default'</em>, <em>autocommit=False</em>, <em>parameters=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/postgres_operator.html#PostgresOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.PostgresOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Executes sql code in a specific Postgres database</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>postgres_conn_id</strong> (<em>string</em>) – reference to a specific postgres database</li>
+<li><strong>sql</strong> (<em>Can receive a str representing a sql statement,
+a list of str (sql statements), or reference to a template file.
+Template reference are recognized by str ending in '.sql'</em>) – the sql code to be executed</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.PrestoCheckOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">PrestoCheckOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>presto_conn_id='presto_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/presto_check_operator.html#PrestoCheckOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.PrestoCheckOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">check_operator.CheckOperator</span></code></p>
+<p>Performs checks against Presto. The <code class="docutils literal"><span class="pre">PrestoCheckOperator</span></code> expects
+a sql query that will return a single row. Each value on that
+first row is evaluated using python <code class="docutils literal"><span class="pre">bool</span></code> casting. If any of the
+values return <code class="docutils literal"><span class="pre">False</span></code> the check is failed and errors out.</p>
+<p>Note that Python bool casting evals the following as <code class="docutils literal"><span class="pre">False</span></code>:
+* False
+* 0
+* Empty string (<code class="docutils literal"><span class="pre">""</span></code>)
+* Empty list (<code class="docutils literal"><span class="pre">[]</span></code>)
+* Empty dictionary or set (<code class="docutils literal"><span class="pre">{}</span></code>)</p>
+<p>Given a query like <code class="docutils literal"><span class="pre">SELECT</span> <span class="pre">COUNT(*)</span> <span class="pre">FROM</span> <span class="pre">foo</span></code>, it will fail only if
+the count <code class="docutils literal"><span class="pre">==</span> <span class="pre">0</span></code>. You can craft much more complex query that could,
+for instance, check that the table has the same number of rows as
+the source table upstream, or that the count of today’s partition is
+greater than yesterday’s partition, or that a set of metrics are less
+than 3 standard deviation for the 7 day average.</p>
+<p>This operator can be used as a data quality check in your pipeline, and
+depending on where you put it in your DAG, you have the choice to
+stop the critical path, preventing from
+publishing dubious data, or on the side and receive email alterts
+without stopping the progress of the DAG.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>sql</strong> (<em>string</em>) – the sql to be executed</li>
+<li><strong>presto_conn_id</strong> (<em>string</em>) – reference to the Presto database</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.PrestoIntervalCheckOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">PrestoIntervalCheckOperator</code><span class="sig-paren">(</span><em>table</em>, <em>metrics_thresholds</em>, <em>date_filter_column='ds'</em>, <em>days_back=-7</em>, <em>presto_conn_id='presto_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/presto_check_operator.html#PrestoIntervalCheckOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.PrestoIntervalCheckOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">check_operator.IntervalCheckOperator</span></code></p>
+<p>Checks that the values of metrics given as SQL expressions are within
+a certain tolerance of the ones from days_back before.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>table</strong> (<em>str</em>) – the table name</li>
+<li><strong>days_back</strong> (<em>int</em>) – number of days between ds and the ds we want to check
+against. Defaults to 7 days</li>
+<li><strong>metrics_threshold</strong> (<em>dict</em>) – a dictionary of ratios indexed by metrics</li>
+<li><strong>presto_conn_id</strong> (<em>string</em>) – reference to the Presto database</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.PrestoValueCheckOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">PrestoValueCheckOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>pass_value</em>, <em>tolerance=None</em>, <em>presto_conn_id='presto_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/presto_check_operator.html#PrestoValueCheckOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.PrestoValueCheckOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">check_operator.ValueCheckOperator</span></code></p>
+<p>Performs a simple value check using sql code.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>sql</strong> (<em>string</em>) – the sql to be executed</li>
+<li><strong>presto_conn_id</strong> (<em>string</em>) – reference to the Presto database</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.PythonOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">PythonOperator</code><span class="sig-paren">(</span><em>python_callable</em>, <em>op_args=None</em>, <em>op_kwargs=None</em>, <em>provide_context=False</em>, <em>templates_dict=None</em>, <em>templates_exts=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/python_operator.html#PythonOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.PythonOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Executes a Python callable</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>python_callable</strong> (<em>python callable</em>) – A reference to an object that is callable</li>
+<li><strong>op_kwargs</strong> (<em>dict</em>) – a dictionary of keyword arguments that will get unpacked
+in your function</li>
+<li><strong>op_args</strong> (<em>list</em>) – a list of positional arguments that will get unpacked when
+calling your callable</li>
+<li><strong>provide_context</strong> (<em>bool</em>) – if set to true, Airflow will pass a set of
+keyword arguments that can be used in your function. This set of
+kwargs correspond exactly to what you can use in your jinja
+templates. For this to work, you need to define <cite>**kwargs</cite> in your
+function header.</li>
+<li><strong>templates_dict</strong> (<em>dict of str</em>) – a dictionary where the values are templates that
+will get templated by the Airflow engine sometime between
+<code class="docutils literal"><span class="pre">__init__</span></code> and <code class="docutils literal"><span class="pre">execute</span></code> takes place and are made available
+in your callable’s context after the template has been applied</li>
+<li><strong>templates_exts</strong> – a list of file extensions to resolve while
+processing templated fields, for examples <code class="docutils literal"><span class="pre">['.sql',</span> <span class="pre">'.hql']</span></code></li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.S3KeySensor">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">S3KeySensor</code><span class="sig-paren">(</span><em>bucket_key</em>, <em>bucket_name=None</em>, <em>wildcard_match=False</em>, <em>s3_conn_id='s3_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#S3KeySensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.S3KeySensor" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.operators.sensors.BaseSensorOperator" title="airflow.operators.sensors.BaseSensorOperator"><code class="xref py py-class docutils literal"><span class="pre">sensors.BaseSensorOperator</span></code></a></p>
+<p>Waits for a key (a file-like instance on S3) to be present in a S3 bucket.
+S3 being a key/value it does not support folders. The path is just a key
+a resource.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>bucket_key</strong> (<em>str</em>) – The key being waited on. Supports full s3:// style url
+or relative path from root level.</li>
+<li><strong>bucket_name</strong> (<em>str</em>) – Name of the S3 bucket</li>
+<li><strong>wildcard_match</strong> (<em>bool</em>) – whether the bucket_key should be interpreted as a
+Unix wildcard pattern</li>
+<li><strong>s3_conn_id</strong> (<em>str</em>) – a reference to the s3 connection</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.S3ToHiveTransfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">S3ToHiveTransfer</code><span class="sig-paren">(</span><em>s3_key</em>, <em>field_dict</em>, <em>hive_table</em>, <em>delimiter='</em>, <em>'</em>, <em>create=True</em>, <em>recreate=False</em>, <em>partition=None</em>, <em>headers=False</em>, <em>check_headers=False</em>, <em>wildcard_match=False</em>, <em>s3_conn_id='s3_default'</em>, <em>hive_cli_conn_id='hive_cli_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/s3_to_hive_operator.html#S3ToHiveTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.S3ToHiveTransfer" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Moves data from S3 to Hive. The operator downloads a file from S3,
+stores the file locally before loading it into a Hive table.
+If the <code class="docutils literal"><span class="pre">create</span></code> or <code class="docutils literal"><span class="pre">recreate</span></code> arguments are set to <code class="docutils literal"><span class="pre">True</span></code>,
+a <code class="docutils literal"><span class="pre">CREATE</span> <span class="pre">TABLE</span></code> and <code class="docutils literal"><span class="pre">DROP</span> <span class="pre">TABLE</span></code> statements are generated.
+Hive data types are inferred from the cursor’s metadata from.</p>
+<p>Note that the table generated in Hive uses <code class="docutils literal"><span class="pre">STORED</span> <span class="pre">AS</span> <span class="pre">textfile</span></code>
+which isn’t the most efficient serialization format. If a
+large amount of data is loaded and/or if the tables gets
+queried considerably, you may want to use this operator only to
+stage the data into a temporary table before loading it into its
+final destination using a <code class="docutils literal"><span class="pre">HiveOperator</span></code>.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>s3_key</strong> (<em>str</em>) – The key to be retrieved from S3</li>
+<li><strong>field_dict</strong> (<em>dict</em>) – A dictionary of the fields name in the file
+as keys and their Hive types as values</li>
+<li><strong>hive_table</strong> (<em>str</em>) – target Hive table, use dot notation to target a
+specific database</li>
+<li><strong>create</strong> (<em>bool</em>) – whether to create the table if it doesn’t exist</li>
+<li><strong>recreate</strong> (<em>bool</em>) – whether to drop and recreate the table at every
+execution</li>
+<li><strong>partition</strong> (<em>dict</em>) – target partition as a dict of partition columns
+and values</li>
+<li><strong>headers</strong> (<em>bool</em>) – whether the file contains column names on the first
+line</li>
+<li><strong>check_headers</strong> (<em>bool</em>) – whether the column names on the first line should be
+checked against the keys of field_dict</li>
+<li><strong>wildcard_match</strong> (<em>bool</em>) – whether the s3_key should be interpreted as a Unix
+wildcard pattern</li>
+<li><strong>delimiter</strong> (<em>str</em>) – field delimiter in the file</li>
+<li><strong>s3_conn_id</strong> (<em>str</em>) – source s3 connection</li>
+<li><strong>hive_conn_id</strong> (<em>str</em>) – destination hive connection</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.ShortCircuitOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">ShortCircuitOperator</code><span class="sig-paren">(</span><em>python_callable</em>, <em>op_args=None</em>, <em>op_kwargs=None</em>, <em>provide_context=False</em>, <em>templates_dict=None</em>, <em>templates_exts=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/python_operator.html#ShortCircuitOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.ShortCircuitOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">python_operator.PythonOperator</span></code></p>
+<p>Allows a workflow to continue only if a condition is met. Otherwise, the
+workflow “short-circuits” and downstream tasks are skipped.</p>
+<p>The ShortCircuitOperator is derived from the PythonOperator. It evaluates a
+condition and short-circuits the workflow if the condition is False. Any
+downstream tasks are marked with a state of “skipped”. If the condition is
+True, downstream tasks proceed as normal.</p>
+<p>The condition is determined by the result of <cite>python_callable</cite>.</p>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.SlackAPIOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">SlackAPIOperator</code><span class="sig-paren">(</span><em>token='unset'</em>, <em>method='unset'</em>, <em>api_params=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/slack_operator.html#SlackAPIOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.SlackAPIOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Base Slack Operator
+The SlackAPIPostOperator is derived from this operator.
+In the future additional Slack API Operators will be derived from this class as well</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>token</strong> (<em>string</em>) – Slack API token (<a class="reference external" href="https://api.slack.com/web">https://api.slack.com/web</a>)</li>
+<li><strong>method</strong> (<em>string</em>) – The Slack API Method to Call (<a class="reference external" href="https://api.slack.com/methods">https://api.slack.com/methods</a>)</li>
+<li><strong>api_params</strong> (<em>dict</em>) – API Method call parameters (<a class="reference external" href="https://api.slack.com/methods">https://api.slack.com/methods</a>)</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+<dl class="method">
+<dt id="airflow.operators.SlackAPIOperator.construct_api_call_params">
+<code class="descname">construct_api_call_params</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/slack_operator.html#SlackAPIOperator.construct_api_call_params"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.SlackAPIOperator.construct_api_call_params" title="Permalink to this definition">�</a></dt>
+<dd><p>Used by the execute function. Allows templating on the source fields of the api_call_params dict before construction</p>
+<p>Override in child classes.
+Each SlackAPIOperator child class is responsible for having a construct_api_call_params function
+which sets self.api_call_params with a dict of API call parameters (<a class="reference external" href="https://api.slack.com/methods">https://api.slack.com/methods</a>)</p>
+</dd></dl>
+
+<dl class="method">
+<dt id="airflow.operators.SlackAPIOperator.execute">
+<code class="descname">execute</code><span class="sig-paren">(</span><em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/slack_operator.html#SlackAPIOperator.execute"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.SlackAPIOperator.execute" title="Permalink to this definition">�</a></dt>
+<dd><p>SlackAPIOperator calls will not fail even if the call is not unsuccessful.
+It should not prevent a DAG from completing in success</p>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.SlackAPIPostOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">SlackAPIPostOperator</code><span class="sig-paren">(</span><em>channel='#general'</em>, <em>username='Airflow'</em>, <em>text='No message has been set.nHere is a cat video insteadnhttps://www.youtube.com/watch?v=J---aiyznGQ'</em>, <em>icon_url='https://raw.githubusercontent.com/airbnb/airflow/master/airflow/www/static/pin_100.png'</em>, <em>attachments=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/slack_operator.html#SlackAPIPostOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.SlackAPIPostOperator" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">slack_operator.SlackAPIOperator</span></code></p>
+<p>Posts messages to a slack channel</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>channel</strong> (<em>string</em>) – channel in which to post message on slack name (#general) or ID (C12318391)</li>
+<li><strong>username</strong> (<em>string</em>) – Username that airflow will be posting to Slack as</li>
+<li><strong>text</strong> (<em>string</em>) – message to send to slack</li>
+<li><strong>icon_url</strong> (<em>string</em>) – url to icon used for this message</li>
+<li><strong>attachments</strong> (<em>array of hashes</em>) – extra formatting details - see <a class="reference external" href="https://api.slack.com/docs/attachments">https://api.slack.com/docs/attachments</a></li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.SqlSensor">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">SqlSensor</code><span class="sig-paren">(</span><em>conn_id</em>, <em>sql</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#SqlSensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.SqlSensor" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.operators.sensors.BaseSensorOperator" title="airflow.operators.sensors.BaseSensorOperator"><code class="xref py py-class docutils literal"><span class="pre">sensors.BaseSensorOperator</span></code></a></p>
+<p>Runs a sql statement until a criteria is met. It will keep trying until
+sql returns no row, or if the first cell in (0, ‘0’, ‘’).</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>conn_id</strong> (<em>string</em>) – The connection to run the sensor against</li>
+<li><strong>sql</strong> – The sql to run. To pass, it needs to return at least one cell
+that contains a non-zero / empty string value.</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.TimeSensor">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">TimeSensor</code><span class="sig-paren">(</span><em>target_time</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#TimeSensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.TimeSensor" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.operators.sensors.BaseSensorOperator" title="airflow.operators.sensors.BaseSensorOperator"><code class="xref py py-class docutils literal"><span class="pre">sensors.BaseSensorOperator</span></code></a></p>
+<p>Waits until the specified time of the day.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>target_time</strong> (<em>datetime.time</em>) – time after which the job succeeds</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.WebHdfsSensor">
+<em class="property">class </em><code class="descclassname">airflow.operators.</code><code class="descname">WebHdfsSensor</code><span class="sig-paren">(</span><em>filepath</em>, <em>webhdfs_conn_id='webhdfs_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/sensors.html#WebHdfsSensor"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.WebHdfsSensor" title="Permalink to this definition">�</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.operators.sensors.BaseSensorOperator" title="airflow.operators.sensors.BaseSensorOperator"><code class="xref py py-class docutils literal"><span class="pre">sensors.BaseSensorOperator</span></code></a></p>
+<p>Waits for a file or folder to land in HDFS</p>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.docker_operator.DockerOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.docker_operator.</code><code class="descname">DockerOperator</code><span class="sig-paren">(</span><em>image</em>, <em>api_version=None</em>, <em>command=None</em>, <em>cpus=1.0</em>, <em>docker_url='unix://var/run/docker.sock'</em>, <em>environment=None</em>, <em>force_pull=False</em>, <em>mem_limit=None</em>, <em>network_mode=None</em>, <em>tls_ca_cert=None</em>, <em>tls_client_cert=None</em>, <em>tls_client_key=None</em>, <em>tls_hostname=None</em>, <em>tls_ssl_version=None</em>, <em>tmp_dir='/tmp/airflow'</em>, <em>user=None</em>, <em>volumes=None</em>, <em>xcom_push=False</em>, <em>xcom_all=False</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/docker_operator.html#DockerOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.docker_operator.DockerOperator" title="Permal
ink to this definition">�</a></dt>
+<dd><p>Execute a command inside a docker container.</p>
+<p>A temporary directory is created on the host and mounted into a container to allow storing files
+that together exceed the default disk size of 10GB in a container. The path to the mounted
+directory can be accessed via the environment variable <code class="docutils literal"><span class="pre">AIRFLOW_TMP_DIR</span></code>.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>image</strong> (<em>str</em>) – Docker image from which to create the container.</li>
<TRUNCATED>
[11/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/jquery-1.11.1.js
----------------------------------------------------------------------
diff --git a/_static/jquery-1.11.1.js b/_static/jquery-1.11.1.js
new file mode 100644
index 0000000..d4b67f7
--- /dev/null
+++ b/_static/jquery-1.11.1.js
@@ -0,0 +1,10308 @@
+/*!
+ * jQuery JavaScript Library v1.11.1
+ * http://jquery.com/
+ *
+ * Includes Sizzle.js
+ * http://sizzlejs.com/
+ *
+ * Copyright 2005, 2014 jQuery Foundation, Inc. and other contributors
+ * Released under the MIT license
+ * http://jquery.org/license
+ *
+ * Date: 2014-05-01T17:42Z
+ */
+
+(function( global, factory ) {
+
+ if ( typeof module === "object" && typeof module.exports === "object" ) {
+ // For CommonJS and CommonJS-like environments where a proper window is present,
+ // execute the factory and get jQuery
+ // For environments that do not inherently posses a window with a document
+ // (such as Node.js), expose a jQuery-making factory as module.exports
+ // This accentuates the need for the creation of a real window
+ // e.g. var jQuery = require("jquery")(window);
+ // See ticket #14549 for more info
+ module.exports = global.document ?
+ factory( global, true ) :
+ function( w ) {
+ if ( !w.document ) {
+ throw new Error( "jQuery requires a window with a document" );
+ }
+ return factory( w );
+ };
+ } else {
+ factory( global );
+ }
+
+// Pass this if window is not defined yet
+}(typeof window !== "undefined" ? window : this, function( window, noGlobal ) {
+
+// Can't do this because several apps including ASP.NET trace
+// the stack via arguments.caller.callee and Firefox dies if
+// you try to trace through "use strict" call chains. (#13335)
+// Support: Firefox 18+
+//
+
+var deletedIds = [];
+
+var slice = deletedIds.slice;
+
+var concat = deletedIds.concat;
+
+var push = deletedIds.push;
+
+var indexOf = deletedIds.indexOf;
+
+var class2type = {};
+
+var toString = class2type.toString;
+
+var hasOwn = class2type.hasOwnProperty;
+
+var support = {};
+
+
+
+var
+ version = "1.11.1",
+
+ // Define a local copy of jQuery
+ jQuery = function( selector, context ) {
+ // The jQuery object is actually just the init constructor 'enhanced'
+ // Need init if jQuery is called (just allow error to be thrown if not included)
+ return new jQuery.fn.init( selector, context );
+ },
+
+ // Support: Android<4.1, IE<9
+ // Make sure we trim BOM and NBSP
+ rtrim = /^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g,
+
+ // Matches dashed string for camelizing
+ rmsPrefix = /^-ms-/,
+ rdashAlpha = /-([\da-z])/gi,
+
+ // Used by jQuery.camelCase as callback to replace()
+ fcamelCase = function( all, letter ) {
+ return letter.toUpperCase();
+ };
+
+jQuery.fn = jQuery.prototype = {
+ // The current version of jQuery being used
+ jquery: version,
+
+ constructor: jQuery,
+
+ // Start with an empty selector
+ selector: "",
+
+ // The default length of a jQuery object is 0
+ length: 0,
+
+ toArray: function() {
+ return slice.call( this );
+ },
+
+ // Get the Nth element in the matched element set OR
+ // Get the whole matched element set as a clean array
+ get: function( num ) {
+ return num != null ?
+
+ // Return just the one element from the set
+ ( num < 0 ? this[ num + this.length ] : this[ num ] ) :
+
+ // Return all the elements in a clean array
+ slice.call( this );
+ },
+
+ // Take an array of elements and push it onto the stack
+ // (returning the new matched element set)
+ pushStack: function( elems ) {
+
+ // Build a new jQuery matched element set
+ var ret = jQuery.merge( this.constructor(), elems );
+
+ // Add the old object onto the stack (as a reference)
+ ret.prevObject = this;
+ ret.context = this.context;
+
+ // Return the newly-formed element set
+ return ret;
+ },
+
+ // Execute a callback for every element in the matched set.
+ // (You can seed the arguments with an array of args, but this is
+ // only used internally.)
+ each: function( callback, args ) {
+ return jQuery.each( this, callback, args );
+ },
+
+ map: function( callback ) {
+ return this.pushStack( jQuery.map(this, function( elem, i ) {
+ return callback.call( elem, i, elem );
+ }));
+ },
+
+ slice: function() {
+ return this.pushStack( slice.apply( this, arguments ) );
+ },
+
+ first: function() {
+ return this.eq( 0 );
+ },
+
+ last: function() {
+ return this.eq( -1 );
+ },
+
+ eq: function( i ) {
+ var len = this.length,
+ j = +i + ( i < 0 ? len : 0 );
+ return this.pushStack( j >= 0 && j < len ? [ this[j] ] : [] );
+ },
+
+ end: function() {
+ return this.prevObject || this.constructor(null);
+ },
+
+ // For internal use only.
+ // Behaves like an Array's method, not like a jQuery method.
+ push: push,
+ sort: deletedIds.sort,
+ splice: deletedIds.splice
+};
+
+jQuery.extend = jQuery.fn.extend = function() {
+ var src, copyIsArray, copy, name, options, clone,
+ target = arguments[0] || {},
+ i = 1,
+ length = arguments.length,
+ deep = false;
+
+ // Handle a deep copy situation
+ if ( typeof target === "boolean" ) {
+ deep = target;
+
+ // skip the boolean and the target
+ target = arguments[ i ] || {};
+ i++;
+ }
+
+ // Handle case when target is a string or something (possible in deep copy)
+ if ( typeof target !== "object" && !jQuery.isFunction(target) ) {
+ target = {};
+ }
+
+ // extend jQuery itself if only one argument is passed
+ if ( i === length ) {
+ target = this;
+ i--;
+ }
+
+ for ( ; i < length; i++ ) {
+ // Only deal with non-null/undefined values
+ if ( (options = arguments[ i ]) != null ) {
+ // Extend the base object
+ for ( name in options ) {
+ src = target[ name ];
+ copy = options[ name ];
+
+ // Prevent never-ending loop
+ if ( target === copy ) {
+ continue;
+ }
+
+ // Recurse if we're merging plain objects or arrays
+ if ( deep && copy && ( jQuery.isPlainObject(copy) || (copyIsArray = jQuery.isArray(copy)) ) ) {
+ if ( copyIsArray ) {
+ copyIsArray = false;
+ clone = src && jQuery.isArray(src) ? src : [];
+
+ } else {
+ clone = src && jQuery.isPlainObject(src) ? src : {};
+ }
+
+ // Never move original objects, clone them
+ target[ name ] = jQuery.extend( deep, clone, copy );
+
+ // Don't bring in undefined values
+ } else if ( copy !== undefined ) {
+ target[ name ] = copy;
+ }
+ }
+ }
+ }
+
+ // Return the modified object
+ return target;
+};
+
+jQuery.extend({
+ // Unique for each copy of jQuery on the page
+ expando: "jQuery" + ( version + Math.random() ).replace( /\D/g, "" ),
+
+ // Assume jQuery is ready without the ready module
+ isReady: true,
+
+ error: function( msg ) {
+ throw new Error( msg );
+ },
+
+ noop: function() {},
+
+ // See test/unit/core.js for details concerning isFunction.
+ // Since version 1.3, DOM methods and functions like alert
+ // aren't supported. They return false on IE (#2968).
+ isFunction: function( obj ) {
+ return jQuery.type(obj) === "function";
+ },
+
+ isArray: Array.isArray || function( obj ) {
+ return jQuery.type(obj) === "array";
+ },
+
+ isWindow: function( obj ) {
+ /* jshint eqeqeq: false */
+ return obj != null && obj == obj.window;
+ },
+
+ isNumeric: function( obj ) {
+ // parseFloat NaNs numeric-cast false positives (null|true|false|"")
+ // ...but misinterprets leading-number strings, particularly hex literals ("0x...")
+ // subtraction forces infinities to NaN
+ return !jQuery.isArray( obj ) && obj - parseFloat( obj ) >= 0;
+ },
+
+ isEmptyObject: function( obj ) {
+ var name;
+ for ( name in obj ) {
+ return false;
+ }
+ return true;
+ },
+
+ isPlainObject: function( obj ) {
+ var key;
+
+ // Must be an Object.
+ // Because of IE, we also have to check the presence of the constructor property.
+ // Make sure that DOM nodes and window objects don't pass through, as well
+ if ( !obj || jQuery.type(obj) !== "object" || obj.nodeType || jQuery.isWindow( obj ) ) {
+ return false;
+ }
+
+ try {
+ // Not own constructor property must be Object
+ if ( obj.constructor &&
+ !hasOwn.call(obj, "constructor") &&
+ !hasOwn.call(obj.constructor.prototype, "isPrototypeOf") ) {
+ return false;
+ }
+ } catch ( e ) {
+ // IE8,9 Will throw exceptions on certain host objects #9897
+ return false;
+ }
+
+ // Support: IE<9
+ // Handle iteration over inherited properties before own properties.
+ if ( support.ownLast ) {
+ for ( key in obj ) {
+ return hasOwn.call( obj, key );
+ }
+ }
+
+ // Own properties are enumerated firstly, so to speed up,
+ // if last one is own, then all properties are own.
+ for ( key in obj ) {}
+
+ return key === undefined || hasOwn.call( obj, key );
+ },
+
+ type: function( obj ) {
+ if ( obj == null ) {
+ return obj + "";
+ }
+ return typeof obj === "object" || typeof obj === "function" ?
+ class2type[ toString.call(obj) ] || "object" :
+ typeof obj;
+ },
+
+ // Evaluates a script in a global context
+ // Workarounds based on findings by Jim Driscoll
+ // http://weblogs.java.net/blog/driscoll/archive/2009/09/08/eval-javascript-global-context
+ globalEval: function( data ) {
+ if ( data && jQuery.trim( data ) ) {
+ // We use execScript on Internet Explorer
+ // We use an anonymous function so that context is window
+ // rather than jQuery in Firefox
+ ( window.execScript || function( data ) {
+ window[ "eval" ].call( window, data );
+ } )( data );
+ }
+ },
+
+ // Convert dashed to camelCase; used by the css and data modules
+ // Microsoft forgot to hump their vendor prefix (#9572)
+ camelCase: function( string ) {
+ return string.replace( rmsPrefix, "ms-" ).replace( rdashAlpha, fcamelCase );
+ },
+
+ nodeName: function( elem, name ) {
+ return elem.nodeName && elem.nodeName.toLowerCase() === name.toLowerCase();
+ },
+
+ // args is for internal usage only
+ each: function( obj, callback, args ) {
+ var value,
+ i = 0,
+ length = obj.length,
+ isArray = isArraylike( obj );
+
+ if ( args ) {
+ if ( isArray ) {
+ for ( ; i < length; i++ ) {
+ value = callback.apply( obj[ i ], args );
+
+ if ( value === false ) {
+ break;
+ }
+ }
+ } else {
+ for ( i in obj ) {
+ value = callback.apply( obj[ i ], args );
+
+ if ( value === false ) {
+ break;
+ }
+ }
+ }
+
+ // A special, fast, case for the most common use of each
+ } else {
+ if ( isArray ) {
+ for ( ; i < length; i++ ) {
+ value = callback.call( obj[ i ], i, obj[ i ] );
+
+ if ( value === false ) {
+ break;
+ }
+ }
+ } else {
+ for ( i in obj ) {
+ value = callback.call( obj[ i ], i, obj[ i ] );
+
+ if ( value === false ) {
+ break;
+ }
+ }
+ }
+ }
+
+ return obj;
+ },
+
+ // Support: Android<4.1, IE<9
+ trim: function( text ) {
+ return text == null ?
+ "" :
+ ( text + "" ).replace( rtrim, "" );
+ },
+
+ // results is for internal usage only
+ makeArray: function( arr, results ) {
+ var ret = results || [];
+
+ if ( arr != null ) {
+ if ( isArraylike( Object(arr) ) ) {
+ jQuery.merge( ret,
+ typeof arr === "string" ?
+ [ arr ] : arr
+ );
+ } else {
+ push.call( ret, arr );
+ }
+ }
+
+ return ret;
+ },
+
+ inArray: function( elem, arr, i ) {
+ var len;
+
+ if ( arr ) {
+ if ( indexOf ) {
+ return indexOf.call( arr, elem, i );
+ }
+
+ len = arr.length;
+ i = i ? i < 0 ? Math.max( 0, len + i ) : i : 0;
+
+ for ( ; i < len; i++ ) {
+ // Skip accessing in sparse arrays
+ if ( i in arr && arr[ i ] === elem ) {
+ return i;
+ }
+ }
+ }
+
+ return -1;
+ },
+
+ merge: function( first, second ) {
+ var len = +second.length,
+ j = 0,
+ i = first.length;
+
+ while ( j < len ) {
+ first[ i++ ] = second[ j++ ];
+ }
+
+ // Support: IE<9
+ // Workaround casting of .length to NaN on otherwise arraylike objects (e.g., NodeLists)
+ if ( len !== len ) {
+ while ( second[j] !== undefined ) {
+ first[ i++ ] = second[ j++ ];
+ }
+ }
+
+ first.length = i;
+
+ return first;
+ },
+
+ grep: function( elems, callback, invert ) {
+ var callbackInverse,
+ matches = [],
+ i = 0,
+ length = elems.length,
+ callbackExpect = !invert;
+
+ // Go through the array, only saving the items
+ // that pass the validator function
+ for ( ; i < length; i++ ) {
+ callbackInverse = !callback( elems[ i ], i );
+ if ( callbackInverse !== callbackExpect ) {
+ matches.push( elems[ i ] );
+ }
+ }
+
+ return matches;
+ },
+
+ // arg is for internal usage only
+ map: function( elems, callback, arg ) {
+ var value,
+ i = 0,
+ length = elems.length,
+ isArray = isArraylike( elems ),
+ ret = [];
+
+ // Go through the array, translating each of the items to their new values
+ if ( isArray ) {
+ for ( ; i < length; i++ ) {
+ value = callback( elems[ i ], i, arg );
+
+ if ( value != null ) {
+ ret.push( value );
+ }
+ }
+
+ // Go through every key on the object,
+ } else {
+ for ( i in elems ) {
+ value = callback( elems[ i ], i, arg );
+
+ if ( value != null ) {
+ ret.push( value );
+ }
+ }
+ }
+
+ // Flatten any nested arrays
+ return concat.apply( [], ret );
+ },
+
+ // A global GUID counter for objects
+ guid: 1,
+
+ // Bind a function to a context, optionally partially applying any
+ // arguments.
+ proxy: function( fn, context ) {
+ var args, proxy, tmp;
+
+ if ( typeof context === "string" ) {
+ tmp = fn[ context ];
+ context = fn;
+ fn = tmp;
+ }
+
+ // Quick check to determine if target is callable, in the spec
+ // this throws a TypeError, but we will just return undefined.
+ if ( !jQuery.isFunction( fn ) ) {
+ return undefined;
+ }
+
+ // Simulated bind
+ args = slice.call( arguments, 2 );
+ proxy = function() {
+ return fn.apply( context || this, args.concat( slice.call( arguments ) ) );
+ };
+
+ // Set the guid of unique handler to the same of original handler, so it can be removed
+ proxy.guid = fn.guid = fn.guid || jQuery.guid++;
+
+ return proxy;
+ },
+
+ now: function() {
+ return +( new Date() );
+ },
+
+ // jQuery.support is not used in Core but other projects attach their
+ // properties to it so it needs to exist.
+ support: support
+});
+
+// Populate the class2type map
+jQuery.each("Boolean Number String Function Array Date RegExp Object Error".split(" "), function(i, name) {
+ class2type[ "[object " + name + "]" ] = name.toLowerCase();
+});
+
+function isArraylike( obj ) {
+ var length = obj.length,
+ type = jQuery.type( obj );
+
+ if ( type === "function" || jQuery.isWindow( obj ) ) {
+ return false;
+ }
+
+ if ( obj.nodeType === 1 && length ) {
+ return true;
+ }
+
+ return type === "array" || length === 0 ||
+ typeof length === "number" && length > 0 && ( length - 1 ) in obj;
+}
+var Sizzle =
+/*!
+ * Sizzle CSS Selector Engine v1.10.19
+ * http://sizzlejs.com/
+ *
+ * Copyright 2013 jQuery Foundation, Inc. and other contributors
+ * Released under the MIT license
+ * http://jquery.org/license
+ *
+ * Date: 2014-04-18
+ */
+(function( window ) {
+
+var i,
+ support,
+ Expr,
+ getText,
+ isXML,
+ tokenize,
+ compile,
+ select,
+ outermostContext,
+ sortInput,
+ hasDuplicate,
+
+ // Local document vars
+ setDocument,
+ document,
+ docElem,
+ documentIsHTML,
+ rbuggyQSA,
+ rbuggyMatches,
+ matches,
+ contains,
+
+ // Instance-specific data
+ expando = "sizzle" + -(new Date()),
+ preferredDoc = window.document,
+ dirruns = 0,
+ done = 0,
+ classCache = createCache(),
+ tokenCache = createCache(),
+ compilerCache = createCache(),
+ sortOrder = function( a, b ) {
+ if ( a === b ) {
+ hasDuplicate = true;
+ }
+ return 0;
+ },
+
+ // General-purpose constants
+ strundefined = typeof undefined,
+ MAX_NEGATIVE = 1 << 31,
+
+ // Instance methods
+ hasOwn = ({}).hasOwnProperty,
+ arr = [],
+ pop = arr.pop,
+ push_native = arr.push,
+ push = arr.push,
+ slice = arr.slice,
+ // Use a stripped-down indexOf if we can't use a native one
+ indexOf = arr.indexOf || function( elem ) {
+ var i = 0,
+ len = this.length;
+ for ( ; i < len; i++ ) {
+ if ( this[i] === elem ) {
+ return i;
+ }
+ }
+ return -1;
+ },
+
+ booleans = "checked|selected|async|autofocus|autoplay|controls|defer|disabled|hidden|ismap|loop|multiple|open|readonly|required|scoped",
+
+ // Regular expressions
+
+ // Whitespace characters http://www.w3.org/TR/css3-selectors/#whitespace
+ whitespace = "[\\x20\\t\\r\\n\\f]",
+ // http://www.w3.org/TR/css3-syntax/#characters
+ characterEncoding = "(?:\\\\.|[\\w-]|[^\\x00-\\xa0])+",
+
+ // Loosely modeled on CSS identifier characters
+ // An unquoted value should be a CSS identifier http://www.w3.org/TR/css3-selectors/#attribute-selectors
+ // Proper syntax: http://www.w3.org/TR/CSS21/syndata.html#value-def-identifier
+ identifier = characterEncoding.replace( "w", "w#" ),
+
+ // Attribute selectors: http://www.w3.org/TR/selectors/#attribute-selectors
+ attributes = "\\[" + whitespace + "*(" + characterEncoding + ")(?:" + whitespace +
+ // Operator (capture 2)
+ "*([*^$|!~]?=)" + whitespace +
+ // "Attribute values must be CSS identifiers [capture 5] or strings [capture 3 or capture 4]"
+ "*(?:'((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\"|(" + identifier + "))|)" + whitespace +
+ "*\\]",
+
+ pseudos = ":(" + characterEncoding + ")(?:\\((" +
+ // To reduce the number of selectors needing tokenize in the preFilter, prefer arguments:
+ // 1. quoted (capture 3; capture 4 or capture 5)
+ "('((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\")|" +
+ // 2. simple (capture 6)
+ "((?:\\\\.|[^\\\\()[\\]]|" + attributes + ")*)|" +
+ // 3. anything else (capture 2)
+ ".*" +
+ ")\\)|)",
+
+ // Leading and non-escaped trailing whitespace, capturing some non-whitespace characters preceding the latter
+ rtrim = new RegExp( "^" + whitespace + "+|((?:^|[^\\\\])(?:\\\\.)*)" + whitespace + "+$", "g" ),
+
+ rcomma = new RegExp( "^" + whitespace + "*," + whitespace + "*" ),
+ rcombinators = new RegExp( "^" + whitespace + "*([>+~]|" + whitespace + ")" + whitespace + "*" ),
+
+ rattributeQuotes = new RegExp( "=" + whitespace + "*([^\\]'\"]*?)" + whitespace + "*\\]", "g" ),
+
+ rpseudo = new RegExp( pseudos ),
+ ridentifier = new RegExp( "^" + identifier + "$" ),
+
+ matchExpr = {
+ "ID": new RegExp( "^#(" + characterEncoding + ")" ),
+ "CLASS": new RegExp( "^\\.(" + characterEncoding + ")" ),
+ "TAG": new RegExp( "^(" + characterEncoding.replace( "w", "w*" ) + ")" ),
+ "ATTR": new RegExp( "^" + attributes ),
+ "PSEUDO": new RegExp( "^" + pseudos ),
+ "CHILD": new RegExp( "^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\(" + whitespace +
+ "*(even|odd|(([+-]|)(\\d*)n|)" + whitespace + "*(?:([+-]|)" + whitespace +
+ "*(\\d+)|))" + whitespace + "*\\)|)", "i" ),
+ "bool": new RegExp( "^(?:" + booleans + ")$", "i" ),
+ // For use in libraries implementing .is()
+ // We use this for POS matching in `select`
+ "needsContext": new RegExp( "^" + whitespace + "*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\(" +
+ whitespace + "*((?:-\\d)?\\d*)" + whitespace + "*\\)|)(?=[^-]|$)", "i" )
+ },
+
+ rinputs = /^(?:input|select|textarea|button)$/i,
+ rheader = /^h\d$/i,
+
+ rnative = /^[^{]+\{\s*\[native \w/,
+
+ // Easily-parseable/retrievable ID or TAG or CLASS selectors
+ rquickExpr = /^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/,
+
+ rsibling = /[+~]/,
+ rescape = /'|\\/g,
+
+ // CSS escapes http://www.w3.org/TR/CSS21/syndata.html#escaped-characters
+ runescape = new RegExp( "\\\\([\\da-f]{1,6}" + whitespace + "?|(" + whitespace + ")|.)", "ig" ),
+ funescape = function( _, escaped, escapedWhitespace ) {
+ var high = "0x" + escaped - 0x10000;
+ // NaN means non-codepoint
+ // Support: Firefox<24
+ // Workaround erroneous numeric interpretation of +"0x"
+ return high !== high || escapedWhitespace ?
+ escaped :
+ high < 0 ?
+ // BMP codepoint
+ String.fromCharCode( high + 0x10000 ) :
+ // Supplemental Plane codepoint (surrogate pair)
+ String.fromCharCode( high >> 10 | 0xD800, high & 0x3FF | 0xDC00 );
+ };
+
+// Optimize for push.apply( _, NodeList )
+try {
+ push.apply(
+ (arr = slice.call( preferredDoc.childNodes )),
+ preferredDoc.childNodes
+ );
+ // Support: Android<4.0
+ // Detect silently failing push.apply
+ arr[ preferredDoc.childNodes.length ].nodeType;
+} catch ( e ) {
+ push = { apply: arr.length ?
+
+ // Leverage slice if possible
+ function( target, els ) {
+ push_native.apply( target, slice.call(els) );
+ } :
+
+ // Support: IE<9
+ // Otherwise append directly
+ function( target, els ) {
+ var j = target.length,
+ i = 0;
+ // Can't trust NodeList.length
+ while ( (target[j++] = els[i++]) ) {}
+ target.length = j - 1;
+ }
+ };
+}
+
+function Sizzle( selector, context, results, seed ) {
+ var match, elem, m, nodeType,
+ // QSA vars
+ i, groups, old, nid, newContext, newSelector;
+
+ if ( ( context ? context.ownerDocument || context : preferredDoc ) !== document ) {
+ setDocument( context );
+ }
+
+ context = context || document;
+ results = results || [];
+
+ if ( !selector || typeof selector !== "string" ) {
+ return results;
+ }
+
+ if ( (nodeType = context.nodeType) !== 1 && nodeType !== 9 ) {
+ return [];
+ }
+
+ if ( documentIsHTML && !seed ) {
+
+ // Shortcuts
+ if ( (match = rquickExpr.exec( selector )) ) {
+ // Speed-up: Sizzle("#ID")
+ if ( (m = match[1]) ) {
+ if ( nodeType === 9 ) {
+ elem = context.getElementById( m );
+ // Check parentNode to catch when Blackberry 4.6 returns
+ // nodes that are no longer in the document (jQuery #6963)
+ if ( elem && elem.parentNode ) {
+ // Handle the case where IE, Opera, and Webkit return items
+ // by name instead of ID
+ if ( elem.id === m ) {
+ results.push( elem );
+ return results;
+ }
+ } else {
+ return results;
+ }
+ } else {
+ // Context is not a document
+ if ( context.ownerDocument && (elem = context.ownerDocument.getElementById( m )) &&
+ contains( context, elem ) && elem.id === m ) {
+ results.push( elem );
+ return results;
+ }
+ }
+
+ // Speed-up: Sizzle("TAG")
+ } else if ( match[2] ) {
+ push.apply( results, context.getElementsByTagName( selector ) );
+ return results;
+
+ // Speed-up: Sizzle(".CLASS")
+ } else if ( (m = match[3]) && support.getElementsByClassName && context.getElementsByClassName ) {
+ push.apply( results, context.getElementsByClassName( m ) );
+ return results;
+ }
+ }
+
+ // QSA path
+ if ( support.qsa && (!rbuggyQSA || !rbuggyQSA.test( selector )) ) {
+ nid = old = expando;
+ newContext = context;
+ newSelector = nodeType === 9 && selector;
+
+ // qSA works strangely on Element-rooted queries
+ // We can work around this by specifying an extra ID on the root
+ // and working up from there (Thanks to Andrew Dupont for the technique)
+ // IE 8 doesn't work on object elements
+ if ( nodeType === 1 && context.nodeName.toLowerCase() !== "object" ) {
+ groups = tokenize( selector );
+
+ if ( (old = context.getAttribute("id")) ) {
+ nid = old.replace( rescape, "\\$&" );
+ } else {
+ context.setAttribute( "id", nid );
+ }
+ nid = "[id='" + nid + "'] ";
+
+ i = groups.length;
+ while ( i-- ) {
+ groups[i] = nid + toSelector( groups[i] );
+ }
+ newContext = rsibling.test( selector ) && testContext( context.parentNode ) || context;
+ newSelector = groups.join(",");
+ }
+
+ if ( newSelector ) {
+ try {
+ push.apply( results,
+ newContext.querySelectorAll( newSelector )
+ );
+ return results;
+ } catch(qsaError) {
+ } finally {
+ if ( !old ) {
+ context.removeAttribute("id");
+ }
+ }
+ }
+ }
+ }
+
+ // All others
+ return select( selector.replace( rtrim, "$1" ), context, results, seed );
+}
+
+/**
+ * Create key-value caches of limited size
+ * @returns {Function(string, Object)} Returns the Object data after storing it on itself with
+ * property name the (space-suffixed) string and (if the cache is larger than Expr.cacheLength)
+ * deleting the oldest entry
+ */
+function createCache() {
+ var keys = [];
+
+ function cache( key, value ) {
+ // Use (key + " ") to avoid collision with native prototype properties (see Issue #157)
+ if ( keys.push( key + " " ) > Expr.cacheLength ) {
+ // Only keep the most recent entries
+ delete cache[ keys.shift() ];
+ }
+ return (cache[ key + " " ] = value);
+ }
+ return cache;
+}
+
+/**
+ * Mark a function for special use by Sizzle
+ * @param {Function} fn The function to mark
+ */
+function markFunction( fn ) {
+ fn[ expando ] = true;
+ return fn;
+}
+
+/**
+ * Support testing using an element
+ * @param {Function} fn Passed the created div and expects a boolean result
+ */
+function assert( fn ) {
+ var div = document.createElement("div");
+
+ try {
+ return !!fn( div );
+ } catch (e) {
+ return false;
+ } finally {
+ // Remove from its parent by default
+ if ( div.parentNode ) {
+ div.parentNode.removeChild( div );
+ }
+ // release memory in IE
+ div = null;
+ }
+}
+
+/**
+ * Adds the same handler for all of the specified attrs
+ * @param {String} attrs Pipe-separated list of attributes
+ * @param {Function} handler The method that will be applied
+ */
+function addHandle( attrs, handler ) {
+ var arr = attrs.split("|"),
+ i = attrs.length;
+
+ while ( i-- ) {
+ Expr.attrHandle[ arr[i] ] = handler;
+ }
+}
+
+/**
+ * Checks document order of two siblings
+ * @param {Element} a
+ * @param {Element} b
+ * @returns {Number} Returns less than 0 if a precedes b, greater than 0 if a follows b
+ */
+function siblingCheck( a, b ) {
+ var cur = b && a,
+ diff = cur && a.nodeType === 1 && b.nodeType === 1 &&
+ ( ~b.sourceIndex || MAX_NEGATIVE ) -
+ ( ~a.sourceIndex || MAX_NEGATIVE );
+
+ // Use IE sourceIndex if available on both nodes
+ if ( diff ) {
+ return diff;
+ }
+
+ // Check if b follows a
+ if ( cur ) {
+ while ( (cur = cur.nextSibling) ) {
+ if ( cur === b ) {
+ return -1;
+ }
+ }
+ }
+
+ return a ? 1 : -1;
+}
+
+/**
+ * Returns a function to use in pseudos for input types
+ * @param {String} type
+ */
+function createInputPseudo( type ) {
+ return function( elem ) {
+ var name = elem.nodeName.toLowerCase();
+ return name === "input" && elem.type === type;
+ };
+}
+
+/**
+ * Returns a function to use in pseudos for buttons
+ * @param {String} type
+ */
+function createButtonPseudo( type ) {
+ return function( elem ) {
+ var name = elem.nodeName.toLowerCase();
+ return (name === "input" || name === "button") && elem.type === type;
+ };
+}
+
+/**
+ * Returns a function to use in pseudos for positionals
+ * @param {Function} fn
+ */
+function createPositionalPseudo( fn ) {
+ return markFunction(function( argument ) {
+ argument = +argument;
+ return markFunction(function( seed, matches ) {
+ var j,
+ matchIndexes = fn( [], seed.length, argument ),
+ i = matchIndexes.length;
+
+ // Match elements found at the specified indexes
+ while ( i-- ) {
+ if ( seed[ (j = matchIndexes[i]) ] ) {
+ seed[j] = !(matches[j] = seed[j]);
+ }
+ }
+ });
+ });
+}
+
+/**
+ * Checks a node for validity as a Sizzle context
+ * @param {Element|Object=} context
+ * @returns {Element|Object|Boolean} The input node if acceptable, otherwise a falsy value
+ */
+function testContext( context ) {
+ return context && typeof context.getElementsByTagName !== strundefined && context;
+}
+
+// Expose support vars for convenience
+support = Sizzle.support = {};
+
+/**
+ * Detects XML nodes
+ * @param {Element|Object} elem An element or a document
+ * @returns {Boolean} True iff elem is a non-HTML XML node
+ */
+isXML = Sizzle.isXML = function( elem ) {
+ // documentElement is verified for cases where it doesn't yet exist
+ // (such as loading iframes in IE - #4833)
+ var documentElement = elem && (elem.ownerDocument || elem).documentElement;
+ return documentElement ? documentElement.nodeName !== "HTML" : false;
+};
+
+/**
+ * Sets document-related variables once based on the current document
+ * @param {Element|Object} [doc] An element or document object to use to set the document
+ * @returns {Object} Returns the current document
+ */
+setDocument = Sizzle.setDocument = function( node ) {
+ var hasCompare,
+ doc = node ? node.ownerDocument || node : preferredDoc,
+ parent = doc.defaultView;
+
+ // If no document and documentElement is available, return
+ if ( doc === document || doc.nodeType !== 9 || !doc.documentElement ) {
+ return document;
+ }
+
+ // Set our document
+ document = doc;
+ docElem = doc.documentElement;
+
+ // Support tests
+ documentIsHTML = !isXML( doc );
+
+ // Support: IE>8
+ // If iframe document is assigned to "document" variable and if iframe has been reloaded,
+ // IE will throw "permission denied" error when accessing "document" variable, see jQuery #13936
+ // IE6-8 do not support the defaultView property so parent will be undefined
+ if ( parent && parent !== parent.top ) {
+ // IE11 does not have attachEvent, so all must suffer
+ if ( parent.addEventListener ) {
+ parent.addEventListener( "unload", function() {
+ setDocument();
+ }, false );
+ } else if ( parent.attachEvent ) {
+ parent.attachEvent( "onunload", function() {
+ setDocument();
+ });
+ }
+ }
+
+ /* Attributes
+ ---------------------------------------------------------------------- */
+
+ // Support: IE<8
+ // Verify that getAttribute really returns attributes and not properties (excepting IE8 booleans)
+ support.attributes = assert(function( div ) {
+ div.className = "i";
+ return !div.getAttribute("className");
+ });
+
+ /* getElement(s)By*
+ ---------------------------------------------------------------------- */
+
+ // Check if getElementsByTagName("*") returns only elements
+ support.getElementsByTagName = assert(function( div ) {
+ div.appendChild( doc.createComment("") );
+ return !div.getElementsByTagName("*").length;
+ });
+
+ // Check if getElementsByClassName can be trusted
+ support.getElementsByClassName = rnative.test( doc.getElementsByClassName ) && assert(function( div ) {
+ div.innerHTML = "<div class='a'></div><div class='a i'></div>";
+
+ // Support: Safari<4
+ // Catch class over-caching
+ div.firstChild.className = "i";
+ // Support: Opera<10
+ // Catch gEBCN failure to find non-leading classes
+ return div.getElementsByClassName("i").length === 2;
+ });
+
+ // Support: IE<10
+ // Check if getElementById returns elements by name
+ // The broken getElementById methods don't pick up programatically-set names,
+ // so use a roundabout getElementsByName test
+ support.getById = assert(function( div ) {
+ docElem.appendChild( div ).id = expando;
+ return !doc.getElementsByName || !doc.getElementsByName( expando ).length;
+ });
+
+ // ID find and filter
+ if ( support.getById ) {
+ Expr.find["ID"] = function( id, context ) {
+ if ( typeof context.getElementById !== strundefined && documentIsHTML ) {
+ var m = context.getElementById( id );
+ // Check parentNode to catch when Blackberry 4.6 returns
+ // nodes that are no longer in the document #6963
+ return m && m.parentNode ? [ m ] : [];
+ }
+ };
+ Expr.filter["ID"] = function( id ) {
+ var attrId = id.replace( runescape, funescape );
+ return function( elem ) {
+ return elem.getAttribute("id") === attrId;
+ };
+ };
+ } else {
+ // Support: IE6/7
+ // getElementById is not reliable as a find shortcut
+ delete Expr.find["ID"];
+
+ Expr.filter["ID"] = function( id ) {
+ var attrId = id.replace( runescape, funescape );
+ return function( elem ) {
+ var node = typeof elem.getAttributeNode !== strundefined && elem.getAttributeNode("id");
+ return node && node.value === attrId;
+ };
+ };
+ }
+
+ // Tag
+ Expr.find["TAG"] = support.getElementsByTagName ?
+ function( tag, context ) {
+ if ( typeof context.getElementsByTagName !== strundefined ) {
+ return context.getElementsByTagName( tag );
+ }
+ } :
+ function( tag, context ) {
+ var elem,
+ tmp = [],
+ i = 0,
+ results = context.getElementsByTagName( tag );
+
+ // Filter out possible comments
+ if ( tag === "*" ) {
+ while ( (elem = results[i++]) ) {
+ if ( elem.nodeType === 1 ) {
+ tmp.push( elem );
+ }
+ }
+
+ return tmp;
+ }
+ return results;
+ };
+
+ // Class
+ Expr.find["CLASS"] = support.getElementsByClassName && function( className, context ) {
+ if ( typeof context.getElementsByClassName !== strundefined && documentIsHTML ) {
+ return context.getElementsByClassName( className );
+ }
+ };
+
+ /* QSA/matchesSelector
+ ---------------------------------------------------------------------- */
+
+ // QSA and matchesSelector support
+
+ // matchesSelector(:active) reports false when true (IE9/Opera 11.5)
+ rbuggyMatches = [];
+
+ // qSa(:focus) reports false when true (Chrome 21)
+ // We allow this because of a bug in IE8/9 that throws an error
+ // whenever `document.activeElement` is accessed on an iframe
+ // So, we allow :focus to pass through QSA all the time to avoid the IE error
+ // See http://bugs.jquery.com/ticket/13378
+ rbuggyQSA = [];
+
+ if ( (support.qsa = rnative.test( doc.querySelectorAll )) ) {
+ // Build QSA regex
+ // Regex strategy adopted from Diego Perini
+ assert(function( div ) {
+ // Select is set to empty string on purpose
+ // This is to test IE's treatment of not explicitly
+ // setting a boolean content attribute,
+ // since its presence should be enough
+ // http://bugs.jquery.com/ticket/12359
+ div.innerHTML = "<select msallowclip=''><option selected=''></option></select>";
+
+ // Support: IE8, Opera 11-12.16
+ // Nothing should be selected when empty strings follow ^= or $= or *=
+ // The test attribute must be unknown in Opera but "safe" for WinRT
+ // http://msdn.microsoft.com/en-us/library/ie/hh465388.aspx#attribute_section
+ if ( div.querySelectorAll("[msallowclip^='']").length ) {
+ rbuggyQSA.push( "[*^$]=" + whitespace + "*(?:''|\"\")" );
+ }
+
+ // Support: IE8
+ // Boolean attributes and "value" are not treated correctly
+ if ( !div.querySelectorAll("[selected]").length ) {
+ rbuggyQSA.push( "\\[" + whitespace + "*(?:value|" + booleans + ")" );
+ }
+
+ // Webkit/Opera - :checked should return selected option elements
+ // http://www.w3.org/TR/2011/REC-css3-selectors-20110929/#checked
+ // IE8 throws error here and will not see later tests
+ if ( !div.querySelectorAll(":checked").length ) {
+ rbuggyQSA.push(":checked");
+ }
+ });
+
+ assert(function( div ) {
+ // Support: Windows 8 Native Apps
+ // The type and name attributes are restricted during .innerHTML assignment
+ var input = doc.createElement("input");
+ input.setAttribute( "type", "hidden" );
+ div.appendChild( input ).setAttribute( "name", "D" );
+
+ // Support: IE8
+ // Enforce case-sensitivity of name attribute
+ if ( div.querySelectorAll("[name=d]").length ) {
+ rbuggyQSA.push( "name" + whitespace + "*[*^$|!~]?=" );
+ }
+
+ // FF 3.5 - :enabled/:disabled and hidden elements (hidden elements are still enabled)
+ // IE8 throws error here and will not see later tests
+ if ( !div.querySelectorAll(":enabled").length ) {
+ rbuggyQSA.push( ":enabled", ":disabled" );
+ }
+
+ // Opera 10-11 does not throw on post-comma invalid pseudos
+ div.querySelectorAll("*,:x");
+ rbuggyQSA.push(",.*:");
+ });
+ }
+
+ if ( (support.matchesSelector = rnative.test( (matches = docElem.matches ||
+ docElem.webkitMatchesSelector ||
+ docElem.mozMatchesSelector ||
+ docElem.oMatchesSelector ||
+ docElem.msMatchesSelector) )) ) {
+
+ assert(function( div ) {
+ // Check to see if it's possible to do matchesSelector
+ // on a disconnected node (IE 9)
+ support.disconnectedMatch = matches.call( div, "div" );
+
+ // This should fail with an exception
+ // Gecko does not error, returns false instead
+ matches.call( div, "[s!='']:x" );
+ rbuggyMatches.push( "!=", pseudos );
+ });
+ }
+
+ rbuggyQSA = rbuggyQSA.length && new RegExp( rbuggyQSA.join("|") );
+ rbuggyMatches = rbuggyMatches.length && new RegExp( rbuggyMatches.join("|") );
+
+ /* Contains
+ ---------------------------------------------------------------------- */
+ hasCompare = rnative.test( docElem.compareDocumentPosition );
+
+ // Element contains another
+ // Purposefully does not implement inclusive descendent
+ // As in, an element does not contain itself
+ contains = hasCompare || rnative.test( docElem.contains ) ?
+ function( a, b ) {
+ var adown = a.nodeType === 9 ? a.documentElement : a,
+ bup = b && b.parentNode;
+ return a === bup || !!( bup && bup.nodeType === 1 && (
+ adown.contains ?
+ adown.contains( bup ) :
+ a.compareDocumentPosition && a.compareDocumentPosition( bup ) & 16
+ ));
+ } :
+ function( a, b ) {
+ if ( b ) {
+ while ( (b = b.parentNode) ) {
+ if ( b === a ) {
+ return true;
+ }
+ }
+ }
+ return false;
+ };
+
+ /* Sorting
+ ---------------------------------------------------------------------- */
+
+ // Document order sorting
+ sortOrder = hasCompare ?
+ function( a, b ) {
+
+ // Flag for duplicate removal
+ if ( a === b ) {
+ hasDuplicate = true;
+ return 0;
+ }
+
+ // Sort on method existence if only one input has compareDocumentPosition
+ var compare = !a.compareDocumentPosition - !b.compareDocumentPosition;
+ if ( compare ) {
+ return compare;
+ }
+
+ // Calculate position if both inputs belong to the same document
+ compare = ( a.ownerDocument || a ) === ( b.ownerDocument || b ) ?
+ a.compareDocumentPosition( b ) :
+
+ // Otherwise we know they are disconnected
+ 1;
+
+ // Disconnected nodes
+ if ( compare & 1 ||
+ (!support.sortDetached && b.compareDocumentPosition( a ) === compare) ) {
+
+ // Choose the first element that is related to our preferred document
+ if ( a === doc || a.ownerDocument === preferredDoc && contains(preferredDoc, a) ) {
+ return -1;
+ }
+ if ( b === doc || b.ownerDocument === preferredDoc && contains(preferredDoc, b) ) {
+ return 1;
+ }
+
+ // Maintain original order
+ return sortInput ?
+ ( indexOf.call( sortInput, a ) - indexOf.call( sortInput, b ) ) :
+ 0;
+ }
+
+ return compare & 4 ? -1 : 1;
+ } :
+ function( a, b ) {
+ // Exit early if the nodes are identical
+ if ( a === b ) {
+ hasDuplicate = true;
+ return 0;
+ }
+
+ var cur,
+ i = 0,
+ aup = a.parentNode,
+ bup = b.parentNode,
+ ap = [ a ],
+ bp = [ b ];
+
+ // Parentless nodes are either documents or disconnected
+ if ( !aup || !bup ) {
+ return a === doc ? -1 :
+ b === doc ? 1 :
+ aup ? -1 :
+ bup ? 1 :
+ sortInput ?
+ ( indexOf.call( sortInput, a ) - indexOf.call( sortInput, b ) ) :
+ 0;
+
+ // If the nodes are siblings, we can do a quick check
+ } else if ( aup === bup ) {
+ return siblingCheck( a, b );
+ }
+
+ // Otherwise we need full lists of their ancestors for comparison
+ cur = a;
+ while ( (cur = cur.parentNode) ) {
+ ap.unshift( cur );
+ }
+ cur = b;
+ while ( (cur = cur.parentNode) ) {
+ bp.unshift( cur );
+ }
+
+ // Walk down the tree looking for a discrepancy
+ while ( ap[i] === bp[i] ) {
+ i++;
+ }
+
+ return i ?
+ // Do a sibling check if the nodes have a common ancestor
+ siblingCheck( ap[i], bp[i] ) :
+
+ // Otherwise nodes in our document sort first
+ ap[i] === preferredDoc ? -1 :
+ bp[i] === preferredDoc ? 1 :
+ 0;
+ };
+
+ return doc;
+};
+
+Sizzle.matches = function( expr, elements ) {
+ return Sizzle( expr, null, null, elements );
+};
+
+Sizzle.matchesSelector = function( elem, expr ) {
+ // Set document vars if needed
+ if ( ( elem.ownerDocument || elem ) !== document ) {
+ setDocument( elem );
+ }
+
+ // Make sure that attribute selectors are quoted
+ expr = expr.replace( rattributeQuotes, "='$1']" );
+
+ if ( support.matchesSelector && documentIsHTML &&
+ ( !rbuggyMatches || !rbuggyMatches.test( expr ) ) &&
+ ( !rbuggyQSA || !rbuggyQSA.test( expr ) ) ) {
+
+ try {
+ var ret = matches.call( elem, expr );
+
+ // IE 9's matchesSelector returns false on disconnected nodes
+ if ( ret || support.disconnectedMatch ||
+ // As well, disconnected nodes are said to be in a document
+ // fragment in IE 9
+ elem.document && elem.document.nodeType !== 11 ) {
+ return ret;
+ }
+ } catch(e) {}
+ }
+
+ return Sizzle( expr, document, null, [ elem ] ).length > 0;
+};
+
+Sizzle.contains = function( context, elem ) {
+ // Set document vars if needed
+ if ( ( context.ownerDocument || context ) !== document ) {
+ setDocument( context );
+ }
+ return contains( context, elem );
+};
+
+Sizzle.attr = function( elem, name ) {
+ // Set document vars if needed
+ if ( ( elem.ownerDocument || elem ) !== document ) {
+ setDocument( elem );
+ }
+
+ var fn = Expr.attrHandle[ name.toLowerCase() ],
+ // Don't get fooled by Object.prototype properties (jQuery #13807)
+ val = fn && hasOwn.call( Expr.attrHandle, name.toLowerCase() ) ?
+ fn( elem, name, !documentIsHTML ) :
+ undefined;
+
+ return val !== undefined ?
+ val :
+ support.attributes || !documentIsHTML ?
+ elem.getAttribute( name ) :
+ (val = elem.getAttributeNode(name)) && val.specified ?
+ val.value :
+ null;
+};
+
+Sizzle.error = function( msg ) {
+ throw new Error( "Syntax error, unrecognized expression: " + msg );
+};
+
+/**
+ * Document sorting and removing duplicates
+ * @param {ArrayLike} results
+ */
+Sizzle.uniqueSort = function( results ) {
+ var elem,
+ duplicates = [],
+ j = 0,
+ i = 0;
+
+ // Unless we *know* we can detect duplicates, assume their presence
+ hasDuplicate = !support.detectDuplicates;
+ sortInput = !support.sortStable && results.slice( 0 );
+ results.sort( sortOrder );
+
+ if ( hasDuplicate ) {
+ while ( (elem = results[i++]) ) {
+ if ( elem === results[ i ] ) {
+ j = duplicates.push( i );
+ }
+ }
+ while ( j-- ) {
+ results.splice( duplicates[ j ], 1 );
+ }
+ }
+
+ // Clear input after sorting to release objects
+ // See https://github.com/jquery/sizzle/pull/225
+ sortInput = null;
+
+ return results;
+};
+
+/**
+ * Utility function for retrieving the text value of an array of DOM nodes
+ * @param {Array|Element} elem
+ */
+getText = Sizzle.getText = function( elem ) {
+ var node,
+ ret = "",
+ i = 0,
+ nodeType = elem.nodeType;
+
+ if ( !nodeType ) {
+ // If no nodeType, this is expected to be an array
+ while ( (node = elem[i++]) ) {
+ // Do not traverse comment nodes
+ ret += getText( node );
+ }
+ } else if ( nodeType === 1 || nodeType === 9 || nodeType === 11 ) {
+ // Use textContent for elements
+ // innerText usage removed for consistency of new lines (jQuery #11153)
+ if ( typeof elem.textContent === "string" ) {
+ return elem.textContent;
+ } else {
+ // Traverse its children
+ for ( elem = elem.firstChild; elem; elem = elem.nextSibling ) {
+ ret += getText( elem );
+ }
+ }
+ } else if ( nodeType === 3 || nodeType === 4 ) {
+ return elem.nodeValue;
+ }
+ // Do not include comment or processing instruction nodes
+
+ return ret;
+};
+
+Expr = Sizzle.selectors = {
+
+ // Can be adjusted by the user
+ cacheLength: 50,
+
+ createPseudo: markFunction,
+
+ match: matchExpr,
+
+ attrHandle: {},
+
+ find: {},
+
+ relative: {
+ ">": { dir: "parentNode", first: true },
+ " ": { dir: "parentNode" },
+ "+": { dir: "previousSibling", first: true },
+ "~": { dir: "previousSibling" }
+ },
+
+ preFilter: {
+ "ATTR": function( match ) {
+ match[1] = match[1].replace( runescape, funescape );
+
+ // Move the given value to match[3] whether quoted or unquoted
+ match[3] = ( match[3] || match[4] || match[5] || "" ).replace( runescape, funescape );
+
+ if ( match[2] === "~=" ) {
+ match[3] = " " + match[3] + " ";
+ }
+
+ return match.slice( 0, 4 );
+ },
+
+ "CHILD": function( match ) {
+ /* matches from matchExpr["CHILD"]
+ 1 type (only|nth|...)
+ 2 what (child|of-type)
+ 3 argument (even|odd|\d*|\d*n([+-]\d+)?|...)
+ 4 xn-component of xn+y argument ([+-]?\d*n|)
+ 5 sign of xn-component
+ 6 x of xn-component
+ 7 sign of y-component
+ 8 y of y-component
+ */
+ match[1] = match[1].toLowerCase();
+
+ if ( match[1].slice( 0, 3 ) === "nth" ) {
+ // nth-* requires argument
+ if ( !match[3] ) {
+ Sizzle.error( match[0] );
+ }
+
+ // numeric x and y parameters for Expr.filter.CHILD
+ // remember that false/true cast respectively to 0/1
+ match[4] = +( match[4] ? match[5] + (match[6] || 1) : 2 * ( match[3] === "even" || match[3] === "odd" ) );
+ match[5] = +( ( match[7] + match[8] ) || match[3] === "odd" );
+
+ // other types prohibit arguments
+ } else if ( match[3] ) {
+ Sizzle.error( match[0] );
+ }
+
+ return match;
+ },
+
+ "PSEUDO": function( match ) {
+ var excess,
+ unquoted = !match[6] && match[2];
+
+ if ( matchExpr["CHILD"].test( match[0] ) ) {
+ return null;
+ }
+
+ // Accept quoted arguments as-is
+ if ( match[3] ) {
+ match[2] = match[4] || match[5] || "";
+
+ // Strip excess characters from unquoted arguments
+ } else if ( unquoted && rpseudo.test( unquoted ) &&
+ // Get excess from tokenize (recursively)
+ (excess = tokenize( unquoted, true )) &&
+ // advance to the next closing parenthesis
+ (excess = unquoted.indexOf( ")", unquoted.length - excess ) - unquoted.length) ) {
+
+ // excess is a negative index
+ match[0] = match[0].slice( 0, excess );
+ match[2] = unquoted.slice( 0, excess );
+ }
+
+ // Return only captures needed by the pseudo filter method (type and argument)
+ return match.slice( 0, 3 );
+ }
+ },
+
+ filter: {
+
+ "TAG": function( nodeNameSelector ) {
+ var nodeName = nodeNameSelector.replace( runescape, funescape ).toLowerCase();
+ return nodeNameSelector === "*" ?
+ function() { return true; } :
+ function( elem ) {
+ return elem.nodeName && elem.nodeName.toLowerCase() === nodeName;
+ };
+ },
+
+ "CLASS": function( className ) {
+ var pattern = classCache[ className + " " ];
+
+ return pattern ||
+ (pattern = new RegExp( "(^|" + whitespace + ")" + className + "(" + whitespace + "|$)" )) &&
+ classCache( className, function( elem ) {
+ return pattern.test( typeof elem.className === "string" && elem.className || typeof elem.getAttribute !== strundefined && elem.getAttribute("class") || "" );
+ });
+ },
+
+ "ATTR": function( name, operator, check ) {
+ return function( elem ) {
+ var result = Sizzle.attr( elem, name );
+
+ if ( result == null ) {
+ return operator === "!=";
+ }
+ if ( !operator ) {
+ return true;
+ }
+
+ result += "";
+
+ return operator === "=" ? result === check :
+ operator === "!=" ? result !== check :
+ operator === "^=" ? check && result.indexOf( check ) === 0 :
+ operator === "*=" ? check && result.indexOf( check ) > -1 :
+ operator === "$=" ? check && result.slice( -check.length ) === check :
+ operator === "~=" ? ( " " + result + " " ).indexOf( check ) > -1 :
+ operator === "|=" ? result === check || result.slice( 0, check.length + 1 ) === check + "-" :
+ false;
+ };
+ },
+
+ "CHILD": function( type, what, argument, first, last ) {
+ var simple = type.slice( 0, 3 ) !== "nth",
+ forward = type.slice( -4 ) !== "last",
+ ofType = what === "of-type";
+
+ return first === 1 && last === 0 ?
+
+ // Shortcut for :nth-*(n)
+ function( elem ) {
+ return !!elem.parentNode;
+ } :
+
+ function( elem, context, xml ) {
+ var cache, outerCache, node, diff, nodeIndex, start,
+ dir = simple !== forward ? "nextSibling" : "previousSibling",
+ parent = elem.parentNode,
+ name = ofType && elem.nodeName.toLowerCase(),
+ useCache = !xml && !ofType;
+
+ if ( parent ) {
+
+ // :(first|last|only)-(child|of-type)
+ if ( simple ) {
+ while ( dir ) {
+ node = elem;
+ while ( (node = node[ dir ]) ) {
+ if ( ofType ? node.nodeName.toLowerCase() === name : node.nodeType === 1 ) {
+ return false;
+ }
+ }
+ // Reverse direction for :only-* (if we haven't yet done so)
+ start = dir = type === "only" && !start && "nextSibling";
+ }
+ return true;
+ }
+
+ start = [ forward ? parent.firstChild : parent.lastChild ];
+
+ // non-xml :nth-child(...) stores cache data on `parent`
+ if ( forward && useCache ) {
+ // Seek `elem` from a previously-cached index
+ outerCache = parent[ expando ] || (parent[ expando ] = {});
+ cache = outerCache[ type ] || [];
+ nodeIndex = cache[0] === dirruns && cache[1];
+ diff = cache[0] === dirruns && cache[2];
+ node = nodeIndex && parent.childNodes[ nodeIndex ];
+
+ while ( (node = ++nodeIndex && node && node[ dir ] ||
+
+ // Fallback to seeking `elem` from the start
+ (diff = nodeIndex = 0) || start.pop()) ) {
+
+ // When found, cache indexes on `parent` and break
+ if ( node.nodeType === 1 && ++diff && node === elem ) {
+ outerCache[ type ] = [ dirruns, nodeIndex, diff ];
+ break;
+ }
+ }
+
+ // Use previously-cached element index if available
+ } else if ( useCache && (cache = (elem[ expando ] || (elem[ expando ] = {}))[ type ]) && cache[0] === dirruns ) {
+ diff = cache[1];
+
+ // xml :nth-child(...) or :nth-last-child(...) or :nth(-last)?-of-type(...)
+ } else {
+ // Use the same loop as above to seek `elem` from the start
+ while ( (node = ++nodeIndex && node && node[ dir ] ||
+ (diff = nodeIndex = 0) || start.pop()) ) {
+
+ if ( ( ofType ? node.nodeName.toLowerCase() === name : node.nodeType === 1 ) && ++diff ) {
+ // Cache the index of each encountered element
+ if ( useCache ) {
+ (node[ expando ] || (node[ expando ] = {}))[ type ] = [ dirruns, diff ];
+ }
+
+ if ( node === elem ) {
+ break;
+ }
+ }
+ }
+ }
+
+ // Incorporate the offset, then check against cycle size
+ diff -= last;
+ return diff === first || ( diff % first === 0 && diff / first >= 0 );
+ }
+ };
+ },
+
+ "PSEUDO": function( pseudo, argument ) {
+ // pseudo-class names are case-insensitive
+ // http://www.w3.org/TR/selectors/#pseudo-classes
+ // Prioritize by case sensitivity in case custom pseudos are added with uppercase letters
+ // Remember that setFilters inherits from pseudos
+ var args,
+ fn = Expr.pseudos[ pseudo ] || Expr.setFilters[ pseudo.toLowerCase() ] ||
+ Sizzle.error( "unsupported pseudo: " + pseudo );
+
+ // The user may use createPseudo to indicate that
+ // arguments are needed to create the filter function
+ // just as Sizzle does
+ if ( fn[ expando ] ) {
+ return fn( argument );
+ }
+
+ // But maintain support for old signatures
+ if ( fn.length > 1 ) {
+ args = [ pseudo, pseudo, "", argument ];
+ return Expr.setFilters.hasOwnProperty( pseudo.toLowerCase() ) ?
+ markFunction(function( seed, matches ) {
+ var idx,
+ matched = fn( seed, argument ),
+ i = matched.length;
+ while ( i-- ) {
+ idx = indexOf.call( seed, matched[i] );
+ seed[ idx ] = !( matches[ idx ] = matched[i] );
+ }
+ }) :
+ function( elem ) {
+ return fn( elem, 0, args );
+ };
+ }
+
+ return fn;
+ }
+ },
+
+ pseudos: {
+ // Potentially complex pseudos
+ "not": markFunction(function( selector ) {
+ // Trim the selector passed to compile
+ // to avoid treating leading and trailing
+ // spaces as combinators
+ var input = [],
+ results = [],
+ matcher = compile( selector.replace( rtrim, "$1" ) );
+
+ return matcher[ expando ] ?
+ markFunction(function( seed, matches, context, xml ) {
+ var elem,
+ unmatched = matcher( seed, null, xml, [] ),
+ i = seed.length;
+
+ // Match elements unmatched by `matcher`
+ while ( i-- ) {
+ if ( (elem = unmatched[i]) ) {
+ seed[i] = !(matches[i] = elem);
+ }
+ }
+ }) :
+ function( elem, context, xml ) {
+ input[0] = elem;
+ matcher( input, null, xml, results );
+ return !results.pop();
+ };
+ }),
+
+ "has": markFunction(function( selector ) {
+ return function( elem ) {
+ return Sizzle( selector, elem ).length > 0;
+ };
+ }),
+
+ "contains": markFunction(function( text ) {
+ return function( elem ) {
+ return ( elem.textContent || elem.innerText || getText( elem ) ).indexOf( text ) > -1;
+ };
+ }),
+
+ // "Whether an element is represented by a :lang() selector
+ // is based solely on the element's language value
+ // being equal to the identifier C,
+ // or beginning with the identifier C immediately followed by "-".
+ // The matching of C against the element's language value is performed case-insensitively.
+ // The identifier C does not have to be a valid language name."
+ // http://www.w3.org/TR/selectors/#lang-pseudo
+ "lang": markFunction( function( lang ) {
+ // lang value must be a valid identifier
+ if ( !ridentifier.test(lang || "") ) {
+ Sizzle.error( "unsupported lang: " + lang );
+ }
+ lang = lang.replace( runescape, funescape ).toLowerCase();
+ return function( elem ) {
+ var elemLang;
+ do {
+ if ( (elemLang = documentIsHTML ?
+ elem.lang :
+ elem.getAttribute("xml:lang") || elem.getAttribute("lang")) ) {
+
+ elemLang = elemLang.toLowerCase();
+ return elemLang === lang || elemLang.indexOf( lang + "-" ) === 0;
+ }
+ } while ( (elem = elem.parentNode) && elem.nodeType === 1 );
+ return false;
+ };
+ }),
+
+ // Miscellaneous
+ "target": function( elem ) {
+ var hash = window.location && window.location.hash;
+ return hash && hash.slice( 1 ) === elem.id;
+ },
+
+ "root": function( elem ) {
+ return elem === docElem;
+ },
+
+ "focus": function( elem ) {
+ return elem === document.activeElement && (!document.hasFocus || document.hasFocus()) && !!(elem.type || elem.href || ~elem.tabIndex);
+ },
+
+ // Boolean properties
+ "enabled": function( elem ) {
+ return elem.disabled === false;
+ },
+
+ "disabled": function( elem ) {
+ return elem.disabled === true;
+ },
+
+ "checked": function( elem ) {
+ // In CSS3, :checked should return both checked and selected elements
+ // http://www.w3.org/TR/2011/REC-css3-selectors-20110929/#checked
+ var nodeName = elem.nodeName.toLowerCase();
+ return (nodeName === "input" && !!elem.checked) || (nodeName === "option" && !!elem.selected);
+ },
+
+ "selected": function( elem ) {
+ // Accessing this property makes selected-by-default
+ // options in Safari work properly
+ if ( elem.parentNode ) {
+ elem.parentNode.selectedIndex;
+ }
+
+ return elem.selected === true;
+ },
+
+ // Contents
+ "empty": function( elem ) {
+ // http://www.w3.org/TR/selectors/#empty-pseudo
+ // :empty is negated by element (1) or content nodes (text: 3; cdata: 4; entity ref: 5),
+ // but not by others (comment: 8; processing instruction: 7; etc.)
+ // nodeType < 6 works because attributes (2) do not appear as children
+ for ( elem = elem.firstChild; elem; elem = elem.nextSibling ) {
+ if ( elem.nodeType < 6 ) {
+ return false;
+ }
+ }
+ return true;
+ },
+
+ "parent": function( elem ) {
+ return !Expr.pseudos["empty"]( elem );
+ },
+
+ // Element/input types
+ "header": function( elem ) {
+ return rheader.test( elem.nodeName );
+ },
+
+ "input": function( elem ) {
+ return rinputs.test( elem.nodeName );
+ },
+
+ "button": function( elem ) {
+ var name = elem.nodeName.toLowerCase();
+ return name === "input" && elem.type === "button" || name === "button";
+ },
+
+ "text": function( elem ) {
+ var attr;
+ return elem.nodeName.toLowerCase() === "input" &&
+ elem.type === "text" &&
+
+ // Support: IE<8
+ // New HTML5 attribute values (e.g., "search") appear with elem.type === "text"
+ ( (attr = elem.getAttribute("type")) == null || attr.toLowerCase() === "text" );
+ },
+
+ // Position-in-collection
+ "first": createPositionalPseudo(function() {
+ return [ 0 ];
+ }),
+
+ "last": createPositionalPseudo(function( matchIndexes, length ) {
+ return [ length - 1 ];
+ }),
+
+ "eq": createPositionalPseudo(function( matchIndexes, length, argument ) {
+ return [ argument < 0 ? argument + length : argument ];
+ }),
+
+ "even": createPositionalPseudo(function( matchIndexes, length ) {
+ var i = 0;
+ for ( ; i < length; i += 2 ) {
+ matchIndexes.push( i );
+ }
+ return matchIndexes;
+ }),
+
+ "odd": createPositionalPseudo(function( matchIndexes, length ) {
+ var i = 1;
+ for ( ; i < length; i += 2 ) {
+ matchIndexes.push( i );
+ }
+ return matchIndexes;
+ }),
+
+ "lt": createPositionalPseudo(function( matchIndexes, length, argument ) {
+ var i = argument < 0 ? argument + length : argument;
+ for ( ; --i >= 0; ) {
+ matchIndexes.push( i );
+ }
+ return matchIndexes;
+ }),
+
+ "gt": createPositionalPseudo(function( matchIndexes, length, argument ) {
+ var i = argument < 0 ? argument + length : argument;
+ for ( ; ++i < length; ) {
+ matchIndexes.push( i );
+ }
+ return matchIndexes;
+ })
+ }
+};
+
+Expr.pseudos["nth"] = Expr.pseudos["eq"];
+
+// Add button/input type pseudos
+for ( i in { radio: true, checkbox: true, file: true, password: true, image: true } ) {
+ Expr.pseudos[ i ] = createInputPseudo( i );
+}
+for ( i in { submit: true, reset: true } ) {
+ Expr.pseudos[ i ] = createButtonPseudo( i );
+}
+
+// Easy API for creating new setFilters
+function setFilters() {}
+setFilters.prototype = Expr.filters = Expr.pseudos;
+Expr.setFilters = new setFilters();
+
+tokenize = Sizzle.tokenize = function( selector, parseOnly ) {
+ var matched, match, tokens, type,
+ soFar, groups, preFilters,
+ cached = tokenCache[ selector + " " ];
+
+ if ( cached ) {
+ return parseOnly ? 0 : cached.slice( 0 );
+ }
+
+ soFar = selector;
+ groups = [];
+ preFilters = Expr.preFilter;
+
+ while ( soFar ) {
+
+ // Comma and first run
+ if ( !matched || (match = rcomma.exec( soFar )) ) {
+ if ( match ) {
+ // Don't consume trailing commas as valid
+ soFar = soFar.slice( match[0].length ) || soFar;
+ }
+ groups.push( (tokens = []) );
+ }
+
+ matched = false;
+
+ // Combinators
+ if ( (match = rcombinators.exec( soFar )) ) {
+ matched = match.shift();
+ tokens.push({
+ value: matched,
+ // Cast descendant combinators to space
+ type: match[0].replace( rtrim, " " )
+ });
+ soFar = soFar.slice( matched.length );
+ }
+
+ // Filters
+ for ( type in Expr.filter ) {
+ if ( (match = matchExpr[ type ].exec( soFar )) && (!preFilters[ type ] ||
+ (match = preFilters[ type ]( match ))) ) {
+ matched = match.shift();
+ tokens.push({
+ value: matched,
+ type: type,
+ matches: match
+ });
+ soFar = soFar.slice( matched.length );
+ }
+ }
+
+ if ( !matched ) {
+ break;
+ }
+ }
+
+ // Return the length of the invalid excess
+ // if we're just parsing
+ // Otherwise, throw an error or return tokens
+ return parseOnly ?
+ soFar.length :
+ soFar ?
+ Sizzle.error( selector ) :
+ // Cache the tokens
+ tokenCache( selector, groups ).slice( 0 );
+};
+
+function toSelector( tokens ) {
+ var i = 0,
+ len = tokens.length,
+ selector = "";
+ for ( ; i < len; i++ ) {
+ selector += tokens[i].value;
+ }
+ return selector;
+}
+
+function addCombinator( matcher, combinator, base ) {
+ var dir = combinator.dir,
+ checkNonElements = base && dir === "parentNode",
+ doneName = done++;
+
+ return combinator.first ?
+ // Check against closest ancestor/preceding element
+ function( elem, context, xml ) {
+ while ( (elem = elem[ dir ]) ) {
+ if ( elem.nodeType === 1 || checkNonElements ) {
+ return matcher( elem, context, xml );
+ }
+ }
+ } :
+
+ // Check against all ancestor/preceding elements
+ function( elem, context, xml ) {
+ var oldCache, outerCache,
+ newCache = [ dirruns, doneName ];
+
+ // We can't set arbitrary data on XML nodes, so they don't benefit from dir caching
+ if ( xml ) {
+ while ( (elem = elem[ dir ]) ) {
+ if ( elem.nodeType === 1 || checkNonElements ) {
+ if ( matcher( elem, context, xml ) ) {
+ return true;
+ }
+ }
+ }
+ } else {
+ while ( (elem = elem[ dir ]) ) {
+ if ( elem.nodeType === 1 || checkNonElements ) {
+ outerCache = elem[ expando ] || (elem[ expando ] = {});
+ if ( (oldCache = outerCache[ dir ]) &&
+ oldCache[ 0 ] === dirruns && oldCache[ 1 ] === doneName ) {
+
+ // Assign to newCache so results back-propagate to previous elements
+ return (newCache[ 2 ] = oldCache[ 2 ]);
+ } else {
+ // Reuse newcache so results back-propagate to previous elements
+ outerCache[ dir ] = newCache;
+
+ // A match means we're done; a fail means we have to keep checking
+ if ( (newCache[ 2 ] = matcher( elem, context, xml )) ) {
+ return true;
+ }
+ }
+ }
+ }
+ }
+ };
+}
+
+function elementMatcher( matchers ) {
+ return matchers.length > 1 ?
+ function( elem, context, xml ) {
+ var i = matchers.length;
+ while ( i-- ) {
+ if ( !matchers[i]( elem, context, xml ) ) {
+ return false;
+ }
+ }
+ return true;
+ } :
+ matchers[0];
+}
+
+function multipleContexts( selector, contexts, results ) {
+ var i = 0,
+ len = contexts.length;
+ for ( ; i < len; i++ ) {
+ Sizzle( selector, contexts[i], results );
+ }
+ return results;
+}
+
+function condense( unmatched, map, filter, context, xml ) {
+ var elem,
+ newUnmatched = [],
+ i = 0,
+ len = unmatched.length,
+ mapped = map != null;
+
+ for ( ; i < len; i++ ) {
+ if ( (elem = unmatched[i]) ) {
+ if ( !filter || filter( elem, context, xml ) ) {
+ newUnmatched.push( elem );
+ if ( mapped ) {
+ map.push( i );
+ }
+ }
+ }
+ }
+
+ return newUnmatched;
+}
+
+function setMatcher( preFilter, selector, matcher, postFilter, postFinder, postSelector ) {
+ if ( postFilter && !postFilter[ expando ] ) {
+ postFilter = setMatcher( postFilter );
+ }
+ if ( postFinder && !postFinder[ expando ] ) {
+ postFinder = setMatcher( postFinder, postSelector );
+ }
+ return markFunction(function( seed, results, context, xml ) {
+ var temp, i, elem,
+ preMap = [],
+ postMap = [],
+ preexisting = results.length,
+
+ // Get initial elements from seed or context
+ elems = seed || multipleContexts( selector || "*", context.nodeType ? [ context ] : context, [] ),
+
+ // Prefilter to get matcher input, preserving a map for seed-results synchronization
+ matcherIn = preFilter && ( seed || !selector ) ?
+ condense( elems, preMap, preFilter, context, xml ) :
+ elems,
+
+ matcherOut = matcher ?
+ // If we have a postFinder, or filtered seed, or non-seed postFilter or preexisting results,
+ postFinder || ( seed ? preFilter : preexisting || postFilter ) ?
+
+ // ...intermediate processing is necessary
+ [] :
+
+ // ...otherwise use results directly
+ results :
+ matcherIn;
+
+ // Find primary matches
+ if ( matcher ) {
+ matcher( matcherIn, matcherOut, context, xml );
+ }
+
+ // Apply postFilter
+ if ( postFilter ) {
+ temp = condense( matcherOut, postMap );
+ postFilter( temp, [], context, xml );
+
+ // Un-match failing elements by moving them back to matcherIn
+ i = temp.length;
+ while ( i-- ) {
+ if ( (elem = temp[i]) ) {
+ matcherOut[ postMap[i] ] = !(matcherIn[ postMap[i] ] = elem);
+ }
+ }
+ }
+
+ if ( seed ) {
+ if ( postFinder || preFilter ) {
+ if ( postFinder ) {
+ // Get the final matcherOut by condensing this intermediate into postFinder contexts
+ temp = [];
+ i = matcherOut.length;
+ while ( i-- ) {
+ if ( (elem = matcherOut[i]) ) {
+ // Restore matcherIn since elem is not yet a final match
+ temp.push( (matcherIn[i] = elem) );
+ }
+ }
+ postFinder( null, (matcherOut = []), temp, xml );
+ }
+
+ // Move matched elements from seed to results to keep them synchronized
+ i = matcherOut.length;
+ while ( i-- ) {
+ if ( (elem = matcherOut[i]) &&
+ (temp = postFinder ? indexOf.call( seed, elem ) : preMap[i]) > -1 ) {
+
+ seed[temp] = !(results[temp] = elem);
+ }
+ }
+ }
+
+ // Add elements to results, through postFinder if defined
+ } else {
+ matcherOut = condense(
+ matcherOut === results ?
+ matcherOut.splice( preexisting, matcherOut.length ) :
+ matcherOut
+ );
+ if ( postFinder ) {
+ postFinder( null, results, matcherOut, xml );
+ } else {
+ push.apply( results, matcherOut );
+ }
+ }
+ });
+}
+
+function matcherFromTokens( tokens ) {
+ var checkContext, matcher, j,
+ len = tokens.length,
+ leadingRelative = Expr.relative[ tokens[0].type ],
+ implicitRelative = leadingRelative || Expr.relative[" "],
+ i = leadingRelative ? 1 : 0,
+
+ // The foundational matcher ensures that elements are reachable from top-level context(s)
+ matchContext = addCombinator( function( elem ) {
+ return elem === checkContext;
+ }, implicitRelative, true ),
+ matchAnyContext = addCombinator( function( elem ) {
+ return indexOf.call( checkContext, elem ) > -1;
+ }, implicitRelative, true ),
+ matchers = [ function( elem, context, xml ) {
+ return ( !leadingRelative && ( xml || context !== outermostContext ) ) || (
+ (checkContext = context).nodeType ?
+ matchContext( elem, context, xml ) :
+ matchAnyContext( elem, context, xml ) );
+ } ];
+
+ for ( ; i < len; i++ ) {
+ if ( (matcher = Expr.relative[ tokens[i].type ]) ) {
+ matchers = [ addCombinator(elementMatcher( matchers ), matcher) ];
+ } else {
+ matcher = Expr.filter[ tokens[i].type ].apply( null, tokens[i].matches );
+
+ // Return special upon seeing a positional matcher
+ if ( matcher[ expando ] ) {
+ // Find the next relative operator (if any) for proper handling
+ j = ++i;
+ for ( ; j < len; j++ ) {
+ if ( Expr.relative[ tokens[j].type ] ) {
+ break;
+ }
+ }
+ return setMatcher(
+ i > 1 && elementMatcher( matchers ),
+ i > 1 && toSelector(
+ // If the preceding token was a descendant combinator, insert an implicit any-element `*`
+ tokens.slice( 0, i - 1 ).concat({ value: tokens[ i - 2 ].type === " " ? "*" : "" })
+ ).replace( rtrim, "$1" ),
+ matcher,
+ i < j && matcherFromTokens( tokens.slice( i, j ) ),
+ j < len && matcherFromTokens( (tokens = tokens.slice( j )) ),
+ j < len && toSelector( tokens )
+ );
+ }
+ matchers.push( matcher );
+ }
+ }
+
+ return elementMatcher( matchers );
+}
+
+function matcherFromGroupMatchers( elementMatchers, setMatchers ) {
+ var bySet = setMatchers.length > 0,
+ byElement = elementMatchers.length > 0,
+ superMatcher = function( seed, context, xml, results, outermost ) {
+ var elem, j, matcher,
+ matchedCount = 0,
+ i = "0",
+ unmatched = seed && [],
+ setMatched = [],
+ contextBackup = outermostContext,
+ // We must always have either seed elements or outermost context
+ elems = seed || byElement && Expr.find["TAG"]( "*", outermost ),
+ // Use integer dirruns iff this is the outermost matcher
+ dirrunsUnique = (dirruns += contextBackup == null ? 1 : Math.random() || 0.1),
+ len = elems.length;
+
+ if ( outermost ) {
+ outermostContext = context !== document && context;
+ }
+
+ // Add elements passing elementMatchers directly to results
+ // Keep `i` a string if there are no elements so `matchedCount` will be "00" below
+ // Support: IE<9, Safari
+ // Tolerate NodeList properties (IE: "length"; Safari: <number>) matching elements by id
+ for ( ; i !== len && (elem = elems[i]) != null; i++ ) {
+ if ( byElement && elem ) {
+ j = 0;
+ while ( (matcher = elementMatchers[j++]) ) {
+ if ( matcher( elem, context, xml ) ) {
+ results.push( elem );
+ break;
+ }
+ }
+ if ( outermost ) {
+ dirruns = dirrunsUnique;
+ }
+ }
+
+ // Track unmatched elements for set filters
+ if ( bySet ) {
+ // They will have gone through all possible matchers
+ if ( (elem = !matcher && elem) ) {
+ matchedCount--;
+ }
+
+ // Lengthen the array for every element, matched or not
+ if ( seed ) {
+ unmatched.push( elem );
+ }
+ }
+ }
+
+ // Apply set filters to unmatched elements
+ matchedCount += i;
+ if ( bySet && i !== matchedCount ) {
+ j = 0;
+ while ( (matcher = setMatchers[j++]) ) {
+ matcher( unmatched, setMatched, context, xml );
+ }
+
+ if ( seed ) {
+ // Reintegrate element matches to eliminate the need for sorting
+ if ( matchedCount > 0 ) {
+ while ( i-- ) {
+ if ( !(unmatched[i] || setMatched[i]) ) {
+ setMatched[i] = pop.call( results );
+ }
+ }
+ }
+
+ // Discard index placeholder values to get only actual matches
+ setMatched = condense( setMatched );
+ }
+
+ // Add matches to results
+ push.apply( results, setMatched );
+
+ // Seedless set matches succeeding multiple successful matchers stipulate sorting
+ if ( outermost && !seed && setMatched.length > 0 &&
+ ( matchedCount + setMatchers.length ) > 1 ) {
+
+ Sizzle.uniqueSort( results );
+ }
+ }
+
+ // Override manipulation of globals by nested matchers
+ if ( outermost ) {
+ dirruns = dirrunsUnique;
+ outermostContext = contextBackup;
+ }
+
+ return unmatched;
+ };
+
+ return bySet ?
+ markFunction( superMatcher ) :
+ superMatcher;
+}
+
+compile = Sizzle.compile = function( selector, match /* Internal Use Only */ ) {
+ var i,
+ setMatchers = [],
+ elementMatchers = [],
+ cached = compilerCache[ selector + " " ];
+
+ if ( !cached ) {
+ // Generate a function of recursive functions that can be used to check each element
+ if ( !match ) {
+ match = tokenize( selector );
+ }
+ i = match.length;
+ while ( i-- ) {
+ cached = matcherFromTokens( match[i] );
+ if ( cached[ expando ] ) {
+ setMatchers.push( cached );
+ } else {
+ elementMatchers.push( cached );
+ }
+ }
+
+ // Cache the compiled function
+ cached = compilerCache( selector, matcherFromGroupMatchers( elementMatchers, setMatchers ) );
+
+ // Save selector and tokenization
+ cached.selector = selector;
+ }
+ return cached;
+};
+
+/**
+ * A low-level selection function that works with Sizzle's compiled
+ * selector functions
+ * @param {String|Function} selector A selector or a pre-compiled
+ * selector function built with Sizzle.compile
+ * @param {Element} context
+ * @param {Array} [results]
+ * @param {Array} [seed] A set of elements to match against
+ */
+select = Sizzle.select = function( selector, context, results, seed ) {
+ var i, tokens, token, type, find,
+ compiled = typeof selector === "function" && selector,
+ match = !seed && tokenize( (selector = compiled.selector || selector) );
+
+ results = results || [];
+
+ // Try to minimize operations if there is no seed and only one group
+ if ( match.length === 1 ) {
+
+ // Take a shortcut and set the context if the root selector is an ID
+ tokens = match[0] = match[0].slice( 0 );
+ if ( tokens.length > 2 && (token = tokens[0]).type === "ID" &&
+ support.getById && context.nodeType === 9 && documentIsHTML &&
+ Expr.relative[ tokens[1].type ] ) {
+
+ context = ( Expr.find["ID"]( token.matches[0].replace(runescape, funescape), context ) || [] )[0];
+ if ( !context ) {
+ return results;
+
+ // Precompiled matchers will still verify ancestry, so step up a level
+ } else if ( compiled ) {
+ context = context.parentNode;
+ }
+
+ selector = selector.slice( tokens.shift().value.length );
+ }
+
+ // Fetch a seed set for right-to-left matching
+ i = matchExpr["needsContext"].test( selector ) ? 0 : tokens.length;
+ while ( i-- ) {
+ token = tokens[i];
+
+ // Abort if we hit a combinator
+ if ( Expr.relative[ (type = token.type) ] ) {
+ break;
+ }
+ if ( (find = Expr.find[ type ]) ) {
+ // Search, expanding context for leading sibling combinators
+ if ( (seed = find(
+ token.matches[0].replace( runescape, funescape ),
+ rsibling.test( tokens[0].type ) && testContext( context.parentNode ) || context
+ )) ) {
+
+ // If seed is empty or no tokens remain, we can return early
+ tokens.splice( i, 1 );
+ selector = seed.length && toSelector( tokens );
+ if ( !selector ) {
+ push.apply( results, seed );
+ return results;
+ }
+
+ break;
+ }
+ }
+ }
+ }
+
+ // Compile and execute a filtering function if one is not provided
+ // Provide `match` to avoid retokenization if we modified the selector above
+ ( compiled || compile( selector, match ) )(
+ seed,
+ context,
+ !documentIsHTML,
+ results,
+ rsibling.test( selector ) && testContext( context.parentNode ) || context
+ );
+ return results;
+};
+
+// One-time assignments
+
+// Sort stability
+support.sortStable = expando.split("").sort( sortOrder ).join("") === expando;
+
+// Support: Chrome<14
+// Always assume duplicates if they aren't passed to the comparison function
+support.detectDuplicates = !!hasDuplicate;
+
+// Initialize against the default document
+setDocument();
+
+// Support: Webkit<537.32 - Safari 6.0.3/Chrome 25 (fixed in Chrome 27)
+// Detached nodes confoundingly follow *each other*
+support.sortDetached = assert(function( div1 ) {
+ // Should return 1, but returns 4 (following)
+ return div1.compareDocumentPosition( document.createElement("div") ) & 1;
+});
+
+// Support: IE<8
+// Prevent attribute/property "interpolation"
+// http://msdn.microsoft.com/en-us/library/ms536429%28VS.85%29.aspx
+if ( !assert(function( div ) {
+ div.innerHTML = "<a href='#'></a>";
+ return div.firstChild.getAttribute("href") === "#" ;
+}) ) {
+ addHandle( "type|href|height|width", function( elem, name, isXML ) {
+ if ( !isXML ) {
+ return elem.getAttribute( name, name.toLowerCase() === "type" ? 1 : 2 );
+ }
+ });
+}
+
+// Support: IE<9
+// Use defaultValue in place of getAttribute("value")
+if ( !support.attributes || !assert(function( div ) {
+ div.innerHTML = "<input/>";
+ div.firstChild.setAttribute( "value", "" );
+ return div.firstChild.getAttribute( "value" ) === "";
+}) ) {
+ addHandle( "value", function( elem, name, isXML ) {
+ if ( !isXML && elem.nodeName.toLowerCase() === "input" ) {
+ return elem.defaultValue;
+ }
+ });
+}
+
+// Support: IE<9
+// Use getAttributeNode to fetch booleans when getAttribute lies
+if ( !assert(function( div ) {
+ return div.getAttribute("disabled") == null;
+}) ) {
+ addHandle( booleans, function( elem, name, isXML ) {
+ var val;
+ if ( !isXML ) {
+ return elem[ name ] === true ? name.toLowerCase() :
+ (val = elem.getAttributeNode( name )) && val.specified ?
+ val.value :
+ null;
+ }
+ });
+}
+
+return Sizzle;
+
+})( window );
+
+
+
+jQuery.find = Sizzle;
+jQuery.expr = Sizzle.selectors;
+jQuery.expr[":"] = jQuery.expr.pseudos;
+jQuery.unique = Sizzle.uniqueSort;
+jQuery.text = Sizzle.getText;
+jQuery.isXMLDoc = Sizzle.isXML;
+jQuery.contains = Sizzle.contains;
+
+
+
+var rneedsContext = jQuery.expr.match.needsContext;
+
+var rsingleTag = (/^<(\w+)\s*\/?>(?:<\/\1>|)$/);
+
+
+
+var risSimple = /^.[^:#\[\.,]*$/;
+
+// Implement the identical functionality for filter and not
+function winnow( elements, qualifier, not ) {
+ if ( jQuery.isFunction( qualifier ) ) {
+ return jQuery.grep( elements, function( elem, i ) {
+ /* jshint -W018 */
+ return !!qualifier.call( elem, i, elem ) !== not;
+ });
+
+ }
+
+ if ( qualifier.nodeType ) {
+ return jQuery.grep( elements, function( elem ) {
+ return ( elem === qualifier ) !== not;
+ });
+
+ }
+
+ if ( typeof qualifier === "string" ) {
+ if ( risSimple.test( qualifier ) ) {
+ return jQuery.filter( qualifier, elements, not );
+ }
+
+ qualifier = jQuery.filter( qualifier, elements );
+ }
+
+ return jQuery.grep( elements, function( elem ) {
+ return ( jQuery.inArray( elem, qualifier ) >= 0 ) !== not;
+ });
+}
+
+jQuery.filter = function( expr, elems, not ) {
+ var elem = elems[ 0 ];
+
+ if ( not ) {
+ expr = ":not(" + expr + ")";
+ }
+
+ return elems.length === 1 && elem.nodeType === 1 ?
+ jQuery.find.matchesSelector( elem, expr ) ? [ elem ] : [] :
+ jQuery.find.matches( expr, jQuery.grep( elems, function( elem ) {
+ return elem.nodeType === 1;
+ }));
+};
+
+jQuery.fn.extend({
+ find: function( selector ) {
+ var i,
+ ret = [],
+ self = this,
+ len = self.length;
+
+ if ( typeof selector !== "string" ) {
+ return this.pushStack( jQuery( selector ).filter(function() {
+ for ( i = 0; i < len; i++ ) {
+ if ( jQuery.contains( self[ i ], this ) ) {
+ return true;
+ }
+ }
+ }) );
+ }
+
+ for ( i = 0; i < len; i++ ) {
+ jQuery.find( selector, self[ i ], ret );
+ }
+
+ // Needed because $( selector, context ) becomes $( context ).find( selector )
+ ret = this.pushStack( len > 1 ? jQuery.unique( ret ) : ret );
+ ret.selector = this.selector ? this.selector + " " + selector : selector;
+ return ret;
+ },
+ filter: function( selector ) {
+ return this.pushStack( winnow(this, selector || [], false) );
+ },
+ not: function( selector ) {
+ return this.pushStack( winnow(this, selector || [], true) );
+ },
+ is: function( selector ) {
+ return !!winnow(
+ this,
+
+ // If this is a positional/relative selector, check membership in the returned set
+ // so $("p:first").is("p:last") won't return true for a doc with two "p".
+ typeof selector === "string" && rneedsContext.test( selector ) ?
+ jQuery( selector ) :
+ selector || [],
+ false
+ ).length;
+ }
+});
+
+
+// Initialize a jQuery object
+
+
+// A central reference to the root jQuery(document)
+var rootjQuery,
+
+ // Use the correct document accordingly with window argument (sandbox)
+ document = window.document,
+
+ // A simple way to check for HTML strings
+ // Prioritize #id over <tag> to avoid XSS via location.hash (#9521)
+ // Strict HTML recognition (#11290: must start with <)
+ rquickExpr = /^(?:\s*(<[\w\W]+>)[^>]*|#([\w-]*))$/,
+
+ init = jQuery.fn.init = function( selector, context ) {
+ var match, elem;
+
+ // HANDLE: $(""), $(null), $(undefined), $(false)
+ if ( !selector ) {
+ return this;
+ }
+
+ // Handle HTML strings
+ if ( typeof selector === "string" ) {
+ if ( selector.charAt(0) === "<" && selector.charAt( selector.length - 1 ) === ">" && selector.length >= 3 ) {
+ // Assume that strings that start and end with <> are HTML and skip the regex check
+ match = [ null, selector, null ];
+
+ } else {
+ match = rquickExpr.exec( selector );
+ }
+
+ // Match html or make sure no context is specified for #id
+ if ( match && (match[1] || !context) ) {
+
+ // HANDLE: $(html) -> $(array)
+ if ( match[1] ) {
+ context = context instanceof jQuery ? context[0] : context;
+
+ // scripts is true for back-compat
+ // Intentionally let the error be thrown if parseHTML is not present
+ jQuery.merge( this, jQuery.parseHTML(
+ match[1],
+ context && context.nodeType ? context.ownerDocument || context : document,
+ true
+ ) );
+
+ // HANDLE: $(html, props)
+ if ( rsingleTag.test( match[1] ) && jQuery.isPlainObject( context ) ) {
+ for ( match in context ) {
+ // Properties of context are called as methods if possible
+ if ( jQuery.isFunction( this[ match ] ) ) {
+ this[ match ]( context[ match ] );
+
+ // ...and otherwise set as attributes
+ } else {
+ this.attr( match, context[ match ] );
+ }
+ }
+ }
+
+ return this;
+
+ // HANDLE: $(#id)
+ } else {
+ elem = document.getElementById( match[2] );
+
+ // Check parentNode to catch when Blackberry 4.6 returns
+ // nodes that are no longer in the document #6963
+ if ( elem && elem.parentNode ) {
+ // Handle the case where IE and Opera return items
+ // by name instead of ID
+ if ( elem.id !== match[2] ) {
+ return rootjQuery.find( selector );
+ }
+
+ // Otherwise, we inject the element directly into the jQuery object
+ this.length = 1;
+ this[0] = elem;
+ }
+
+ this.context = document;
+ this.selector = selector;
+ return this;
+ }
+
+ // HANDLE: $(expr, $(...))
+ } else if ( !context || context.jquery ) {
+ return ( context || rootjQuery ).find( selector );
+
+ // HANDLE: $(expr, context)
+ // (which is just equivalent to: $(context).find(expr)
+ } else {
+ return this.constructor( context ).find( selector );
+ }
+
+ // HANDLE: $(DOMElement)
+ } else if ( selector.nodeType ) {
+ this.context = this[0] = selector;
+ this.length = 1;
+ return this;
+
+ // HANDLE: $(function)
+ // Shortcut for document ready
+ } else if ( jQuery.isFunction( selector ) ) {
+ return typeof rootjQuery.ready !== "undefined" ?
+ rootjQuery.ready( selector ) :
+ // Execute immediately if ready is not present
+ selector( jQuery );
+ }
+
+ if ( selector.selector !== undefined ) {
+ this.selector = selector.selector;
+ this.context = selector.context;
+ }
+
+ return jQuery.makeArray( selector, this );
+ };
+
+// Give the init function the jQuery prototype for later instantiation
+init.prototype = jQuery.fn;
+
+// Initialize central reference
+rootjQuery = jQuery( document );
+
+
+var rparentsprev = /^(?:parents|prev(?:Until|All))/,
+ // methods guaranteed to produce a unique set when starting from a unique set
+ guaranteedUnique = {
+ children: true,
+ contents: true,
+ next: true,
+ prev: true
+ };
+
+jQuery.extend({
+ dir: function( elem, dir, until ) {
+ var matched = [],
+ cur = elem[ dir ];
+
+ while ( cur && cur.nodeType !== 9 && (until === undefined || cur.nodeType !== 1 || !jQuery( cur ).is( until )) ) {
+ if ( cur.nodeType === 1 ) {
+ matched.push( cur );
+ }
+ cur = cur[dir];
+ }
+ return matched;
+ },
+
+ sibling: function( n, elem ) {
+ var r = [];
+
+ for ( ; n; n = n.nextSibling ) {
+ if ( n.nodeType === 1 && n !== elem ) {
+ r.push( n );
+ }
+ }
+
+ return r;
+ }
+});
+
+jQuery.fn.extend({
+ has: function( target ) {
+ var i,
+ targets = jQuery( target, this ),
+ len = targets.length;
+
+ return this.filter(function() {
+ for ( i = 0; i < len; i++ ) {
+ if ( jQuery.contains( this, targets[i] ) ) {
+ return true;
+ }
+ }
+ });
+ },
+
+ closest: function( selectors, context ) {
+ var cur,
+ i = 0,
+ l = this.length,
+ matched = [],
+ pos = rneedsContext.test( selectors ) || typeof selectors !== "string" ?
+ jQuery( selectors, context || this.context ) :
+ 0;
+
+ for ( ; i < l; i++ ) {
+ for ( cur = this[i]; cur && cur !== context; cur = cur.parentNode ) {
+ // Always skip document fragments
+ if ( cur.nodeType < 11 && (pos ?
+ pos.index(cur) > -1 :
+
+ // Don't pass non-elements to Sizzle
+ cur.nodeType === 1 &&
+ jQuery.find.matchesSelector(cur, selectors)) ) {
+
+ matched.push( cur );
+ break;
+ }
+ }
+ }
+
+ return this.pushStack( matched.length > 1 ? jQuery.unique( matched ) : matched );
+ },
+
+ // Determine the position of an element within
+ // the matched set of elements
+ index: function( elem ) {
+
+ // No argument, return index in parent
+ if ( !elem ) {
+ return ( this[0] && this[0].parentNode ) ? this.first().prevAll().length : -1;
+ }
+
+ // index in selector
+ if ( typeof elem === "string" ) {
+ return jQuery.inArray( this[0], jQuery( elem ) );
+ }
+
+ // Locate the position of the desired element
+ return jQuery.inArray(
+ // If it receives a jQuery object, the first element is used
+ elem.jquery ? elem[0] : elem, this );
+ },
+
+ add: function( selector, context ) {
+ return this.pushStack(
+ jQuery.unique(
+ jQuery.merge( this.get(), jQuery( selector, context ) )
+ )
+ );
+ },
+
+ addBack: function( selector ) {
+ return this.add( selector == null ?
+ this.prevObject : this.prevObject.filter(selector)
+ );
+ }
+});
+
+function sibling( cur, dir ) {
+ do {
+ cur = cur[ dir ];
+ } while ( cur && cur.nodeType !== 1 );
+
+ return cur;
+}
+
+jQuery.each({
+ parent: function( elem ) {
+ var parent = elem.parentNode;
+ return parent && parent.nodeType !== 11 ? parent : null;
+ },
+ parents: function( elem ) {
+ return jQuery.dir( elem, "parentNode" );
+ },
+ parentsUntil: function( elem, i, until ) {
+ return jQuery.dir( elem, "parentNode", until );
+ },
+ next: function( elem ) {
+ return sibling( elem, "nextSibling" );
+ },
+ prev: function( elem ) {
+ return sibling( elem, "previousSibling" );
+ },
+ nextAll: function( elem ) {
+ return jQuery.dir( elem, "nextSibling" );
+ },
+ prevAll: function( elem ) {
+ return jQuery.dir( elem, "previousSibling" );
+ },
+ nextUntil: function( elem, i, until ) {
+ return jQuery.dir( elem, "nextSibling", until );
+ },
+ prevUntil: function( elem, i, until ) {
+ return jQuery.dir( elem, "previousSibling", until );
+ },
+ siblings: function( elem ) {
+ return jQuery.sibling( ( elem.parentNode || {} ).firstChild, elem );
+ },
+ children: function( elem ) {
+ return jQuery.sibling( elem.firstChild );
+ },
+ contents: function( elem ) {
+ return jQuery.nodeName( elem, "iframe" ) ?
+ elem.contentDocument || elem.contentWindow.document :
+ jQuery.merge( [], elem.childNodes );
+ }
+}, function( name, fn ) {
+ jQuery.fn[ name ] = function( until, selector ) {
+ var ret = jQuery.map( this, fn, until );
+
+ if ( name.slice( -5 ) !== "Until" ) {
+ selector = until;
+ }
+
+ if ( selector && typeof selector === "string" ) {
+ ret = jQuery.filter( selector, ret );
+ }
+
+ if ( this.length > 1 ) {
+ // Remove duplicates
+ if ( !guaranteedUnique[ name ] ) {
+ ret = jQuery.unique( ret );
+ }
+
+ // Reverse order for parents* and prev-derivatives
+ if ( rparentsprev.test( name ) ) {
+ ret = ret.reverse();
+ }
+ }
+
+ return this.pushStack( ret );
+ };
+});
+var rnotwhite = (/\S+/g);
+
+
+
+// String to Object options format cache
+var optionsCache = {};
+
+// Convert String-formatted options into Object-formatted ones and store in cache
+function createOptions( options ) {
+ var object = optionsCache[ options ] = {};
+ jQuery.each( options.match( rnotwhite ) || [], function( _, flag ) {
+ object[ flag ] = true;
+ });
+ return object;
+}
+
+/*
+ * Create a callback list using the following parameters:
+ *
+ * options: an optional list of space-separated options that will change how
+ * the callback list behaves or a more traditional option object
+ *
+ * By default a callback list will act like an event callback list and can be
+ * "fired" multiple times.
+ *
+ * Possible options:
+ *
+ * once: will ensure the callback list can only be fired once (like a Deferred)
+ *
+ * memory: will keep track of previous values and will call any callback added
+ * after the list has been fired right away with the latest "memorized"
+ * values (like a Deferred)
+ *
+ * unique: will ensure a callback can only be added once (no duplicate in the list)
+ *
+ * stopOnFalse: interrupt callings when a callback returns false
+ *
+ */
+jQuery.Callbacks = function( options ) {
+
+ // Convert options from String-formatted to Object-formatted if needed
+ // (we check in cache first)
+ options = typeof options === "string" ?
+ ( optionsCache[ options ] || createOptions( options ) ) :
+ jQuery.extend( {}, options );
+
+ var // Flag to know if list is currently firing
+ firing,
+ // Last fire value (for non-forgettable lists)
+ memory,
+ // Flag to know if list was already fired
+ fired,
+ // End of the loop when firing
+ firingLength,
+ // Index of currently firing callback (modified by remove if needed)
+ firingIndex,
+ // First callback to fire (used internally by add and fireWith)
+ firingStart,
+ // Actual callback list
+ list = [],
+ // Stack of fire calls for repeatable lists
+ stack = !options.once && [],
+ // Fire callbacks
+ fire = function( data ) {
+ memory = options.memory && data;
+ fired = true;
+ firingIndex = firingStart || 0;
+ firingStart = 0;
+ firingLength = list.length;
+ firing = true;
+ for ( ; list && firingIndex < firingLength; firingIndex++ ) {
+ if ( list[ firingIndex ].apply( data[ 0 ], data[ 1 ] ) === false && options.stopOnFalse ) {
+ memory = false; // To prevent further calls using add
+ break;
+ }
+ }
+ firing = false;
+ if ( list ) {
+ if ( stack ) {
+ if ( stack.length ) {
+ fire( stack.shift() );
+ }
+ } else if ( memory ) {
+ list = [];
+ } else {
+ self.disable();
+ }
+ }
+ },
+ // Actual Callbacks object
+ self = {
+ // Add a callback or a collection of callbacks to the list
+ add: function() {
+ if ( list ) {
+ // First, we save the current length
+ var start = list.length;
+ (function add( args ) {
+ jQuery.each( args, function( _, arg ) {
+ var type = jQuery.type( arg );
+ if ( type === "function" ) {
+ if ( !options.unique || !self.has( arg ) ) {
+ list.push( arg );
+ }
+ } else if ( arg && arg.length && type !== "string" ) {
+ // Inspect recursively
+
<TRUNCATED>
[20/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/sensors.html
----------------------------------------------------------------------
diff --git a/_modules/sensors.html b/_modules/sensors.html
new file mode 100644
index 0000000..7f4b8ec
--- /dev/null
+++ b/_modules/sensors.html
@@ -0,0 +1,721 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>sensors — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>sensors</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for sensors</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">print_function</span>
+<span class="kn">from</span> <span class="nn">future</span> <span class="kn">import</span> <span class="n">standard_library</span>
+<span class="n">standard_library</span><span class="o">.</span><span class="n">install_aliases</span><span class="p">()</span>
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">str</span>
+<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">from</span> <span class="nn">urllib.parse</span> <span class="kn">import</span> <span class="n">urlparse</span>
+<span class="kn">from</span> <span class="nn">time</span> <span class="kn">import</span> <span class="n">sleep</span>
+
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="kn">import</span> <span class="n">hooks</span><span class="p">,</span> <span class="n">settings</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span><span class="p">,</span> <span class="n">AirflowSensorTimeout</span><span class="p">,</span> <span class="n">AirflowSkipException</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span><span class="p">,</span> <span class="n">TaskInstance</span><span class="p">,</span> <span class="n">Connection</span> <span class="k">as</span> <span class="n">DB</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">BaseHook</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.state</span> <span class="kn">import</span> <span class="n">State</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<span class="k">class</span> <span class="nc">BaseSensorOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> Sensor operators are derived from this class an inherit these attributes.</span>
+
+<span class="sd"> Sensor operators keep executing at a time interval and succeed when</span>
+<span class="sd"> a criteria is met and fail if and when they time out.</span>
+
+<span class="sd"> :param soft_fail: Set to true to mark the task as SKIPPED on failure</span>
+<span class="sd"> :type soft_fail: bool</span>
+<span class="sd"> :param poke_interval: Time in seconds that the job should wait in</span>
+<span class="sd"> between each tries</span>
+<span class="sd"> :type poke_interval: int</span>
+<span class="sd"> :param timeout: Time, in seconds before the task times out and fails.</span>
+<span class="sd"> :type timeout: int</span>
+<span class="sd"> '''</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#e6f1f2'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">poke_interval</span><span class="o">=</span><span class="mi">60</span><span class="p">,</span>
+ <span class="n">timeout</span><span class="o">=</span><span class="mi">60</span><span class="o">*</span><span class="mi">60</span><span class="o">*</span><span class="mi">24</span><span class="o">*</span><span class="mi">7</span><span class="p">,</span>
+ <span class="n">soft_fail</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">poke_interval</span> <span class="o">=</span> <span class="n">poke_interval</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">soft_fail</span> <span class="o">=</span> <span class="n">soft_fail</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">timeout</span> <span class="o">=</span> <span class="n">timeout</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> Function that the sensors defined while deriving this class should</span>
+<span class="sd"> override.</span>
+<span class="sd"> '''</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Override me.'</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">started_at</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+ <span class="k">while</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">poke</span><span class="p">(</span><span class="n">context</span><span class="p">):</span>
+ <span class="n">sleep</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">poke_interval</span><span class="p">)</span>
+ <span class="k">if</span> <span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">-</span> <span class="n">started_at</span><span class="p">)</span><span class="o">.</span><span class="n">seconds</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">timeout</span><span class="p">:</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">soft_fail</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowSkipException</span><span class="p">(</span><span class="s1">'Snap. Time is OUT.'</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowSensorTimeout</span><span class="p">(</span><span class="s1">'Snap. Time is OUT.'</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Success criteria met. Exiting."</span><span class="p">)</span>
+
+
+<div class="viewcode-block" id="SqlSensor"><a class="viewcode-back" href="../code.html#airflow.operators.SqlSensor">[docs]</a><span class="k">class</span> <span class="nc">SqlSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Runs a sql statement until a criteria is met. It will keep trying until</span>
+<span class="sd"> sql returns no row, or if the first cell in (0, '0', '').</span>
+
+<span class="sd"> :param conn_id: The connection to run the sensor against</span>
+<span class="sd"> :type conn_id: string</span>
+<span class="sd"> :param sql: The sql to run. To pass, it needs to return at least one cell</span>
+<span class="sd"> that contains a non-zero / empty string value.</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.hql'</span><span class="p">,</span> <span class="s1">'.sql'</span><span class="p">,)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">conn_id</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span> <span class="o">=</span> <span class="n">conn_id</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">BaseHook</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span><span class="p">)</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span>
+
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking: '</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+ <span class="n">records</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">get_records</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">records</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">False</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">if</span> <span class="nb">str</span><span class="p">(</span><span class="n">records</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">])</span> <span class="ow">in</span> <span class="p">(</span><span class="s1">'0'</span><span class="p">,</span> <span class="s1">''</span><span class="p">,):</span>
+ <span class="k">return</span> <span class="bp">False</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">True</span>
+ <span class="k">print</span><span class="p">(</span><span class="n">records</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">])</span></div>
+
+
+<div class="viewcode-block" id="MetastorePartitionSensor"><a class="viewcode-back" href="../code.html#airflow.operators.MetastorePartitionSensor">[docs]</a><span class="k">class</span> <span class="nc">MetastorePartitionSensor</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> An alternative to the HivePartitionSensor that talk directly to the</span>
+<span class="sd"> MySQL db. This was created as a result of observing sub optimal</span>
+<span class="sd"> queries generated by the Metastore thrift service when hitting</span>
+<span class="sd"> subpartitioned tables. The Thrift service's queries were written in a</span>
+<span class="sd"> way that wouldn't leverage the indexes.</span>
+
+<span class="sd"> :param schema: the schema</span>
+<span class="sd"> :type schema: str</span>
+<span class="sd"> :param table: the table</span>
+<span class="sd"> :type table: str</span>
+<span class="sd"> :param partition_name: the partition name, as defined in the PARTITIONS</span>
+<span class="sd"> table of the Metastore. Order of the fields does matter.</span>
+<span class="sd"> Examples: ``ds=2016-01-01`` or</span>
+<span class="sd"> ``ds=2016-01-01/sub=foo`` for a sub partitioned table</span>
+<span class="sd"> :type partition_name: str</span>
+<span class="sd"> :param mysql_conn_id: a reference to the MySQL conn_id for the metastore</span>
+<span class="sd"> :type mysql_conn_id: str</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'partition_name'</span><span class="p">,</span> <span class="s1">'table'</span><span class="p">,</span> <span class="s1">'schema'</span><span class="p">)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">partition_name</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="s2">"default"</span><span class="p">,</span>
+ <span class="n">mysql_conn_id</span><span class="o">=</span><span class="s2">"metastore_mysql"</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">partition_name</span> <span class="o">=</span> <span class="n">partition_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">table</span> <span class="o">=</span> <span class="n">table</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="n">schema</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">first_poke</span> <span class="o">=</span> <span class="bp">True</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span> <span class="o">=</span> <span class="n">mysql_conn_id</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">first_poke</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">first_poke</span> <span class="o">=</span> <span class="bp">False</span>
+ <span class="k">if</span> <span class="s1">'.'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="s2">"""</span>
+<span class="s2"> SELECT 'X'</span>
+<span class="s2"> FROM PARTITIONS A0</span>
+<span class="s2"> LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID = B0.TBL_ID</span>
+<span class="s2"> LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID</span>
+<span class="s2"> WHERE</span>
+<span class="s2"> B0.TBL_NAME = '{self.table}' AND</span>
+<span class="s2"> C0.NAME = '{self.schema}' AND</span>
+<span class="s2"> A0.PART_NAME = '{self.partition_name}';</span>
+<span class="s2"> """</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">=</span><span class="bp">self</span><span class="p">)</span>
+ <span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">MetastorePartitionSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">poke</span><span class="p">(</span><span class="n">context</span><span class="p">)</span></div>
+
+
+<div class="viewcode-block" id="ExternalTaskSensor"><a class="viewcode-back" href="../code.html#airflow.operators.ExternalTaskSensor">[docs]</a><span class="k">class</span> <span class="nc">ExternalTaskSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for a task to complete in a different DAG</span>
+
+<span class="sd"> :param external_dag_id: The dag_id that contains the task you want to</span>
+<span class="sd"> wait for</span>
+<span class="sd"> :type external_dag_id: string</span>
+<span class="sd"> :param external_task_id: The task_id that contains the task you want to</span>
+<span class="sd"> wait for</span>
+<span class="sd"> :type external_task_id: string</span>
+<span class="sd"> :param allowed_states: list of allowed states, default is ``['success']``</span>
+<span class="sd"> :type allowed_states: list</span>
+<span class="sd"> :param execution_delta: time difference with the previous execution to</span>
+<span class="sd"> look at, the default is the same execution_date as the current task.</span>
+<span class="sd"> For yesterday, use [positive!] datetime.timedelta(days=1)</span>
+<span class="sd"> :type execution_delta: datetime.timedelta</span>
+<span class="sd"> """</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">external_dag_id</span><span class="p">,</span>
+ <span class="n">external_task_id</span><span class="p">,</span>
+ <span class="n">allowed_states</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">execution_delta</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">ExternalTaskSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">allowed_states</span> <span class="o">=</span> <span class="n">allowed_states</span> <span class="ow">or</span> <span class="p">[</span><span class="n">State</span><span class="o">.</span><span class="n">SUCCESS</span><span class="p">]</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">execution_delta</span> <span class="o">=</span> <span class="n">execution_delta</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">external_dag_id</span> <span class="o">=</span> <span class="n">external_dag_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">external_task_id</span> <span class="o">=</span> <span class="n">external_task_id</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">execution_delta</span><span class="p">:</span>
+ <span class="n">dttm</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'execution_date'</span><span class="p">]</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">execution_delta</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">dttm</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'execution_date'</span><span class="p">]</span>
+
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Poking for '</span>
+ <span class="s1">'{self.external_dag_id}.'</span>
+ <span class="s1">'{self.external_task_id} on '</span>
+ <span class="s1">'{dttm} ... '</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="n">TI</span> <span class="o">=</span> <span class="n">TaskInstance</span>
+
+ <span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
+ <span class="n">count</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">TI</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span>
+ <span class="n">TI</span><span class="o">.</span><span class="n">dag_id</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">external_dag_id</span><span class="p">,</span>
+ <span class="n">TI</span><span class="o">.</span><span class="n">task_id</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">external_task_id</span><span class="p">,</span>
+ <span class="n">TI</span><span class="o">.</span><span class="n">state</span><span class="o">.</span><span class="n">in_</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">allowed_states</span><span class="p">),</span>
+ <span class="n">TI</span><span class="o">.</span><span class="n">execution_date</span> <span class="o">==</span> <span class="n">dttm</span><span class="p">,</span>
+ <span class="p">)</span><span class="o">.</span><span class="n">count</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">count</span></div>
+
+
+<div class="viewcode-block" id="HivePartitionSensor"><a class="viewcode-back" href="../code.html#airflow.operators.HivePartitionSensor">[docs]</a><span class="k">class</span> <span class="nc">HivePartitionSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for a partition to show up in Hive</span>
+
+<span class="sd"> :param table: The name of the table to wait for, supports the dot</span>
+<span class="sd"> notation (my_database.my_table)</span>
+<span class="sd"> :type table: string</span>
+<span class="sd"> :param partition: The partition clause to wait for. This is passed as</span>
+<span class="sd"> is to the Metastore Thrift client "get_partitions_by_filter" method,</span>
+<span class="sd"> and apparently supports SQL like notation as in `ds='2015-01-01'</span>
+<span class="sd"> AND type='value'` and > < sings as in "ds>=2015-01-01"</span>
+<span class="sd"> :type partition: string</span>
+<span class="sd"> :param metastore_conn_id: reference to the metastore thrift service</span>
+<span class="sd"> connection id</span>
+<span class="sd"> :type metastore_conn_id: str</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'schema'</span><span class="p">,</span> <span class="s1">'table'</span><span class="p">,</span> <span class="s1">'partition'</span><span class="p">,)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">table</span><span class="p">,</span> <span class="n">partition</span><span class="o">=</span><span class="s2">"ds='{{ ds }}'"</span><span class="p">,</span>
+ <span class="n">metastore_conn_id</span><span class="o">=</span><span class="s1">'metastore_default'</span><span class="p">,</span>
+ <span class="n">schema</span><span class="o">=</span><span class="s1">'default'</span><span class="p">,</span>
+ <span class="n">poke_interval</span><span class="o">=</span><span class="mi">60</span><span class="o">*</span><span class="mi">3</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HivePartitionSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span>
+ <span class="n">poke_interval</span><span class="o">=</span><span class="n">poke_interval</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">partition</span><span class="p">:</span>
+ <span class="n">partition</span> <span class="o">=</span> <span class="s2">"ds='{{ ds }}'"</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metastore_conn_id</span> <span class="o">=</span> <span class="n">metastore_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">table</span> <span class="o">=</span> <span class="n">table</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">partition</span> <span class="o">=</span> <span class="n">partition</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="n">schema</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="k">if</span> <span class="s1">'.'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Poking for table {self.schema}.{self.table}, '</span>
+ <span class="s1">'partition {self.partition}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s1">'hook'</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">HiveMetastoreHook</span><span class="p">(</span>
+ <span class="n">metastore_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">metastore_conn_id</span><span class="p">)</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">check_for_partition</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">partition</span><span class="p">)</span></div>
+
+
+<div class="viewcode-block" id="HdfsSensor"><a class="viewcode-back" href="../code.html#airflow.operators.HdfsSensor">[docs]</a><span class="k">class</span> <span class="nc">HdfsSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for a file or folder to land in HDFS</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'filepath'</span><span class="p">,)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">filepath</span><span class="p">,</span>
+ <span class="n">hdfs_conn_id</span><span class="o">=</span><span class="s1">'hdfs_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HdfsSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">filepath</span> <span class="o">=</span> <span class="n">filepath</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hdfs_conn_id</span> <span class="o">=</span> <span class="n">hdfs_conn_id</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">sb</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">HDFSHook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">hdfs_conn_id</span><span class="p">)</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s2">"snakebite"</span><span class="p">)</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">WARNING</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Poking for file {self.filepath} '</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">files</span> <span class="o">=</span> <span class="p">[</span><span class="n">f</span> <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">sb</span><span class="o">.</span><span class="n">ls</span><span class="p">([</span><span class="bp">self</span><span class="o">.</span><span class="n">filepath</span><span class="p">])]</span>
+ <span class="k">except</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">False</span>
+ <span class="k">return</span> <span class="bp">True</span></div>
+
+
+<div class="viewcode-block" id="WebHdfsSensor"><a class="viewcode-back" href="../code.html#airflow.operators.WebHdfsSensor">[docs]</a><span class="k">class</span> <span class="nc">WebHdfsSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for a file or folder to land in HDFS</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'filepath'</span><span class="p">,)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">filepath</span><span class="p">,</span>
+ <span class="n">webhdfs_conn_id</span><span class="o">=</span><span class="s1">'webhdfs_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">WebHdfsSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">filepath</span> <span class="o">=</span> <span class="n">filepath</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">webhdfs_conn_id</span> <span class="o">=</span> <span class="n">webhdfs_conn_id</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">c</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">WebHDFSHook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">webhdfs_conn_id</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Poking for file {self.filepath} '</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="k">return</span> <span class="n">c</span><span class="o">.</span><span class="n">check_for_path</span><span class="p">(</span><span class="n">hdfs_path</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">filepath</span><span class="p">)</span></div>
+
+
+<div class="viewcode-block" id="S3KeySensor"><a class="viewcode-back" href="../code.html#airflow.operators.S3KeySensor">[docs]</a><span class="k">class</span> <span class="nc">S3KeySensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for a key (a file-like instance on S3) to be present in a S3 bucket.</span>
+<span class="sd"> S3 being a key/value it does not support folders. The path is just a key</span>
+<span class="sd"> a resource.</span>
+
+<span class="sd"> :param bucket_key: The key being waited on. Supports full s3:// style url</span>
+<span class="sd"> or relative path from root level.</span>
+<span class="sd"> :type bucket_key: str</span>
+<span class="sd"> :param bucket_name: Name of the S3 bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> :param wildcard_match: whether the bucket_key should be interpreted as a</span>
+<span class="sd"> Unix wildcard pattern</span>
+<span class="sd"> :type wildcard_match: bool</span>
+<span class="sd"> :param s3_conn_id: a reference to the s3 connection</span>
+<span class="sd"> :type s3_conn_id: str</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'bucket_key'</span><span class="p">,</span> <span class="s1">'bucket_name'</span><span class="p">)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">bucket_key</span><span class="p">,</span>
+ <span class="n">bucket_name</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">wildcard_match</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">s3_conn_id</span><span class="o">=</span><span class="s1">'s3_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">S3KeySensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
+ <span class="n">db</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">DB</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">DB</span><span class="o">.</span><span class="n">conn_id</span> <span class="o">==</span> <span class="n">s3_conn_id</span><span class="p">)</span><span class="o">.</span><span class="n">first</span><span class="p">()</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">db</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"conn_id doesn't exist in the repository"</span><span class="p">)</span>
+ <span class="c1"># Parse</span>
+ <span class="k">if</span> <span class="n">bucket_name</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
+ <span class="n">parsed_url</span> <span class="o">=</span> <span class="n">urlparse</span><span class="p">(</span><span class="n">bucket_key</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">netloc</span> <span class="o">==</span> <span class="s1">''</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Please provide a bucket_name'</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">bucket_name</span> <span class="o">=</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">netloc</span>
+ <span class="k">if</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">path</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'/'</span><span class="p">:</span>
+ <span class="n">bucket_key</span> <span class="o">=</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">path</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">bucket_key</span> <span class="o">=</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">path</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span> <span class="o">=</span> <span class="n">bucket_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bucket_key</span> <span class="o">=</span> <span class="n">bucket_key</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">wildcard_match</span> <span class="o">=</span> <span class="n">wildcard_match</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span> <span class="o">=</span> <span class="n">s3_conn_id</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">S3Hook</span><span class="p">(</span><span class="n">s3_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span><span class="p">)</span>
+ <span class="n">full_url</span> <span class="o">=</span> <span class="s2">"s3://"</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span> <span class="o">+</span> <span class="s2">"/"</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">bucket_key</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for key : {full_url}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">wildcard_match</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">check_for_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">bucket_key</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">check_for_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">bucket_key</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span><span class="p">)</span></div>
+
+
+<span class="k">class</span> <span class="nc">S3PrefixSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for a prefix to exist. A prefix is the first part of a key,</span>
+<span class="sd"> thus enabling checking of constructs similar to glob airfl* or</span>
+<span class="sd"> SQL LIKE 'airfl%'. There is the possibility to precise a delimiter to</span>
+<span class="sd"> indicate the hierarchy or keys, meaning that the match will stop at that</span>
+<span class="sd"> delimiter. Current code accepts sane delimiters, i.e. characters that</span>
+<span class="sd"> are NOT special characters in the Python regex engine.</span>
+
+<span class="sd"> :param bucket_name: Name of the S3 bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> :param prefix: The prefix being waited on. Relative path from bucket root level.</span>
+<span class="sd"> :type prefix: str</span>
+<span class="sd"> :param delimiter: The delimiter intended to show hierarchy.</span>
+<span class="sd"> Defaults to '/'.</span>
+<span class="sd"> :type delimiter: str</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'prefix'</span><span class="p">,</span> <span class="s1">'bucket_name'</span><span class="p">)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span>
+ <span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">'/'</span><span class="p">,</span>
+ <span class="n">s3_conn_id</span><span class="o">=</span><span class="s1">'s3_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">S3PrefixSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
+ <span class="n">db</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">DB</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">DB</span><span class="o">.</span><span class="n">conn_id</span> <span class="o">==</span> <span class="n">s3_conn_id</span><span class="p">)</span><span class="o">.</span><span class="n">first</span><span class="p">()</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">db</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"conn_id doesn't exist in the repository"</span><span class="p">)</span>
+ <span class="c1"># Parse</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span> <span class="o">=</span> <span class="n">bucket_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">prefix</span> <span class="o">=</span> <span class="n">prefix</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span> <span class="o">=</span> <span class="n">delimiter</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">full_url</span> <span class="o">=</span> <span class="s2">"s3://"</span> <span class="o">+</span> <span class="n">bucket_name</span> <span class="o">+</span> <span class="s1">'/'</span> <span class="o">+</span> <span class="n">prefix</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span> <span class="o">=</span> <span class="n">s3_conn_id</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for prefix : {self.prefix}</span><span class="se">\n</span><span class="s1">'</span>
+ <span class="s1">'in bucket s3://{self.bucket_name}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">S3Hook</span><span class="p">(</span><span class="n">s3_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">check_for_prefix</span><span class="p">(</span>
+ <span class="n">prefix</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">prefix</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span>
+ <span class="n">bucket_name</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span><span class="p">)</span>
+
+
+<div class="viewcode-block" id="TimeSensor"><a class="viewcode-back" href="../code.html#airflow.operators.TimeSensor">[docs]</a><span class="k">class</span> <span class="nc">TimeSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits until the specified time of the day.</span>
+
+<span class="sd"> :param target_time: time after which the job succeeds</span>
+<span class="sd"> :type target_time: datetime.time</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">()</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">target_time</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">TimeSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">target_time</span> <span class="o">=</span> <span class="n">target_time</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Checking if the time ({0}) has come'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">target_time</span><span class="p">))</span>
+ <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">target_time</span></div>
+
+
+<span class="k">class</span> <span class="nc">TimeDeltaSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for a timedelta after the task's execution_date + schedule_interval.</span>
+<span class="sd"> In Airflow, the daily task stamped with ``execution_date``</span>
+<span class="sd"> 2016-01-01 can only start running on 2016-01-02. The timedelta here</span>
+<span class="sd"> represents the time after the execution period has closed.</span>
+
+<span class="sd"> :param delta: time length to wait after execution_date before succeeding</span>
+<span class="sd"> :type delta: datetime.timedelta</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">()</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">delta</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">TimeDeltaSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delta</span> <span class="o">=</span> <span class="n">delta</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">dag</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'dag'</span><span class="p">]</span>
+ <span class="n">target_dttm</span> <span class="o">=</span> <span class="n">dag</span><span class="o">.</span><span class="n">following_schedule</span><span class="p">(</span><span class="n">context</span><span class="p">[</span><span class="s1">'execution_date'</span><span class="p">])</span>
+ <span class="n">target_dttm</span> <span class="o">+=</span> <span class="bp">self</span><span class="o">.</span><span class="n">delta</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Checking if the time ({0}) has come'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">target_dttm</span><span class="p">))</span>
+ <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">></span> <span class="n">target_dttm</span>
+
+
+<div class="viewcode-block" id="HttpSensor"><a class="viewcode-back" href="../code.html#airflow.operators.HttpSensor">[docs]</a><span class="k">class</span> <span class="nc">HttpSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes a HTTP get statement and returns False on failure:</span>
+<span class="sd"> 404 not found or response_check function returned False</span>
+
+<span class="sd"> :param http_conn_id: The connection to run the sensor against</span>
+<span class="sd"> :type http_conn_id: string</span>
+<span class="sd"> :param endpoint: The relative part of the full url</span>
+<span class="sd"> :type endpoint: string</span>
+<span class="sd"> :param params: The parameters to be added to the GET url</span>
+<span class="sd"> :type params: a dictionary of string key/value pairs</span>
+<span class="sd"> :param headers: The HTTP headers to be added to the GET request</span>
+<span class="sd"> :type headers: a dictionary of string key/value pairs</span>
+<span class="sd"> :param response_check: A check against the 'requests' response object.</span>
+<span class="sd"> Returns True for 'pass' and False otherwise.</span>
+<span class="sd"> :type response_check: A lambda or defined function.</span>
+<span class="sd"> :param extra_options: Extra options for the 'requests' library, see the</span>
+<span class="sd"> 'requests' documentation (options to modify timeout, ssl, etc.)</span>
+<span class="sd"> :type extra_options: A dictionary of options, where key is string and value</span>
+<span class="sd"> depends on the option that's being modified.</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'endpoint'</span><span class="p">,)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">endpoint</span><span class="p">,</span>
+ <span class="n">http_conn_id</span><span class="o">=</span><span class="s1">'http_default'</span><span class="p">,</span>
+ <span class="n">params</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">headers</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">response_check</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">extra_options</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HttpSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span> <span class="o">=</span> <span class="n">endpoint</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">http_conn_id</span> <span class="o">=</span> <span class="n">http_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">params</span> <span class="o">=</span> <span class="n">params</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">headers</span> <span class="o">=</span> <span class="n">headers</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">extra_options</span> <span class="o">=</span> <span class="n">extra_options</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">response_check</span> <span class="o">=</span> <span class="n">response_check</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">HttpHook</span><span class="p">(</span><span class="n">method</span><span class="o">=</span><span class="s1">'GET'</span><span class="p">,</span> <span class="n">http_conn_id</span><span class="o">=</span><span class="n">http_conn_id</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking: '</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span><span class="p">)</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span><span class="p">,</span>
+ <span class="n">data</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">params</span><span class="p">,</span>
+ <span class="n">headers</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">headers</span><span class="p">,</span>
+ <span class="n">extra_options</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">extra_options</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">response_check</span><span class="p">:</span>
+ <span class="c1"># run content check on response</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">response_check</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
+ <span class="k">except</span> <span class="n">AirflowException</span> <span class="k">as</span> <span class="n">ae</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">ae</span><span class="o">.</span><span class="n">message</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"404"</span><span class="p">):</span>
+ <span class="k">return</span> <span class="bp">False</span>
+
+ <span class="k">raise</span> <span class="n">ae</span>
+
+ <span class="k">return</span> <span class="bp">True</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[31/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/airflow/models.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/models.html b/_modules/airflow/models.html
new file mode 100644
index 0000000..9ec98be
--- /dev/null
+++ b/_modules/airflow/models.html
@@ -0,0 +1,3802 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.models — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../../index.html"/>
+ <link rel="up" title="Module code" href="../index.html"/>
+
+
+ <script src="../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../../index.html">Docs</a> »</li>
+
+ <li><a href="../index.html">Module code</a> »</li>
+
+ <li>airflow.models</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.models</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+<span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">absolute_import</span>
+<span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">division</span>
+<span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">print_function</span>
+<span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">unicode_literals</span>
+
+<span class="kn">from</span> <span class="nn">future.standard_library</span> <span class="kn">import</span> <span class="n">install_aliases</span>
+
+<span class="n">install_aliases</span><span class="p">()</span>
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">str</span>
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">object</span><span class="p">,</span> <span class="nb">bytes</span>
+<span class="kn">import</span> <span class="nn">copy</span>
+<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">namedtuple</span>
+<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timedelta</span>
+<span class="kn">import</span> <span class="nn">dill</span>
+<span class="kn">import</span> <span class="nn">functools</span>
+<span class="kn">import</span> <span class="nn">getpass</span>
+<span class="kn">import</span> <span class="nn">imp</span>
+<span class="kn">import</span> <span class="nn">importlib</span>
+<span class="kn">import</span> <span class="nn">zipfile</span>
+<span class="kn">import</span> <span class="nn">jinja2</span>
+<span class="kn">import</span> <span class="nn">json</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">import</span> <span class="nn">os</span>
+<span class="kn">import</span> <span class="nn">pickle</span>
+<span class="kn">import</span> <span class="nn">re</span>
+<span class="kn">import</span> <span class="nn">signal</span>
+<span class="kn">import</span> <span class="nn">socket</span>
+<span class="kn">import</span> <span class="nn">sys</span>
+<span class="kn">import</span> <span class="nn">textwrap</span>
+<span class="kn">import</span> <span class="nn">traceback</span>
+<span class="kn">import</span> <span class="nn">warnings</span>
+<span class="kn">from</span> <span class="nn">urllib.parse</span> <span class="kn">import</span> <span class="n">urlparse</span>
+
+<span class="kn">from</span> <span class="nn">sqlalchemy</span> <span class="kn">import</span> <span class="p">(</span>
+ <span class="n">Column</span><span class="p">,</span> <span class="n">Integer</span><span class="p">,</span> <span class="n">String</span><span class="p">,</span> <span class="n">DateTime</span><span class="p">,</span> <span class="n">Text</span><span class="p">,</span> <span class="n">Boolean</span><span class="p">,</span> <span class="n">ForeignKey</span><span class="p">,</span> <span class="n">PickleType</span><span class="p">,</span>
+ <span class="n">Index</span><span class="p">,</span> <span class="n">Float</span><span class="p">)</span>
+<span class="kn">from</span> <span class="nn">sqlalchemy</span> <span class="kn">import</span> <span class="n">case</span><span class="p">,</span> <span class="n">func</span><span class="p">,</span> <span class="n">or_</span><span class="p">,</span> <span class="n">and_</span>
+<span class="kn">from</span> <span class="nn">sqlalchemy.ext.declarative</span> <span class="kn">import</span> <span class="n">declarative_base</span><span class="p">,</span> <span class="n">declared_attr</span>
+<span class="kn">from</span> <span class="nn">sqlalchemy.dialects.mysql</span> <span class="kn">import</span> <span class="n">LONGTEXT</span>
+<span class="kn">from</span> <span class="nn">sqlalchemy.orm</span> <span class="kn">import</span> <span class="n">relationship</span><span class="p">,</span> <span class="n">synonym</span>
+
+<span class="kn">from</span> <span class="nn">croniter</span> <span class="kn">import</span> <span class="n">croniter</span>
+<span class="kn">import</span> <span class="nn">six</span>
+
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="kn">import</span> <span class="n">settings</span><span class="p">,</span> <span class="n">utils</span>
+<span class="kn">from</span> <span class="nn">airflow.executors</span> <span class="kn">import</span> <span class="n">DEFAULT_EXECUTOR</span><span class="p">,</span> <span class="n">LocalExecutor</span>
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="kn">import</span> <span class="n">configuration</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span><span class="p">,</span> <span class="n">AirflowSkipException</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.dates</span> <span class="kn">import</span> <span class="n">cron_presets</span><span class="p">,</span> <span class="n">date_range</span> <span class="k">as</span> <span class="n">utils_date_range</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.db</span> <span class="kn">import</span> <span class="n">provide_session</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.email</span> <span class="kn">import</span> <span class="n">send_email</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.helpers</span> <span class="kn">import</span> <span class="p">(</span>
+ <span class="n">as_tuple</span><span class="p">,</span> <span class="n">is_container</span><span class="p">,</span> <span class="n">is_in</span><span class="p">,</span> <span class="n">validate_key</span><span class="p">,</span> <span class="n">pprinttable</span><span class="p">)</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.logging</span> <span class="kn">import</span> <span class="n">LoggingMixin</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.state</span> <span class="kn">import</span> <span class="n">State</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.timeout</span> <span class="kn">import</span> <span class="n">timeout</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.trigger_rule</span> <span class="kn">import</span> <span class="n">TriggerRule</span>
+
+<span class="n">Base</span> <span class="o">=</span> <span class="n">declarative_base</span><span class="p">()</span>
+<span class="n">ID_LEN</span> <span class="o">=</span> <span class="mi">250</span>
+<span class="n">SQL_ALCHEMY_CONN</span> <span class="o">=</span> <span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'SQL_ALCHEMY_CONN'</span><span class="p">)</span>
+<span class="n">DAGS_FOLDER</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">expanduser</span><span class="p">(</span><span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'DAGS_FOLDER'</span><span class="p">))</span>
+<span class="n">XCOM_RETURN_KEY</span> <span class="o">=</span> <span class="s1">'return_value'</span>
+
+<span class="n">Stats</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Stats</span>
+
+<span class="n">ENCRYPTION_ON</span> <span class="o">=</span> <span class="bp">False</span>
+<span class="k">try</span><span class="p">:</span>
+ <span class="kn">from</span> <span class="nn">cryptography.fernet</span> <span class="kn">import</span> <span class="n">Fernet</span>
+ <span class="n">FERNET</span> <span class="o">=</span> <span class="n">Fernet</span><span class="p">(</span><span class="n">configuration</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'FERNET_KEY'</span><span class="p">)</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s1">'utf-8'</span><span class="p">))</span>
+ <span class="n">ENCRYPTION_ON</span> <span class="o">=</span> <span class="bp">True</span>
+<span class="k">except</span><span class="p">:</span>
+ <span class="k">pass</span>
+
+<span class="k">if</span> <span class="s1">'mysql'</span> <span class="ow">in</span> <span class="n">SQL_ALCHEMY_CONN</span><span class="p">:</span>
+ <span class="n">LongText</span> <span class="o">=</span> <span class="n">LONGTEXT</span>
+<span class="k">else</span><span class="p">:</span>
+ <span class="n">LongText</span> <span class="o">=</span> <span class="n">Text</span>
+
+<span class="c1"># used by DAG context_managers</span>
+<span class="n">_CONTEXT_MANAGER_DAG</span> <span class="o">=</span> <span class="bp">None</span>
+
+
+<span class="k">def</span> <span class="nf">clear_task_instances</span><span class="p">(</span><span class="n">tis</span><span class="p">,</span> <span class="n">session</span><span class="p">,</span> <span class="n">activate_dag_runs</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> Clears a set of task instances, but makes sure the running ones</span>
+<span class="sd"> get killed.</span>
+<span class="sd"> '''</span>
+ <span class="n">job_ids</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="k">for</span> <span class="n">ti</span> <span class="ow">in</span> <span class="n">tis</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">ti</span><span class="o">.</span><span class="n">state</span> <span class="o">==</span> <span class="n">State</span><span class="o">.</span><span class="n">RUNNING</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">ti</span><span class="o">.</span><span class="n">job_id</span><span class="p">:</span>
+ <span class="n">ti</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">State</span><span class="o">.</span><span class="n">SHUTDOWN</span>
+ <span class="n">job_ids</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">ti</span><span class="o">.</span><span class="n">job_id</span><span class="p">)</span>
+ <span class="c1"># todo: this creates an issue with the webui tests</span>
+ <span class="c1">#elif ti.state != State.REMOVED:</span>
+ <span class="c1"># ti.state = State.NONE</span>
+ <span class="c1"># session.merge(ti)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="n">ti</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">job_ids</span><span class="p">:</span>
+ <span class="kn">from</span> <span class="nn">airflow.jobs</span> <span class="kn">import</span> <span class="n">BaseJob</span> <span class="k">as</span> <span class="n">BJ</span>
+ <span class="k">for</span> <span class="n">job</span> <span class="ow">in</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">BJ</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">BJ</span><span class="o">.</span><span class="n">id</span><span class="o">.</span><span class="n">in_</span><span class="p">(</span><span class="n">job_ids</span><span class="p">))</span><span class="o">.</span><span class="n">all</span><span class="p">():</span>
+ <span class="n">job</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">State</span><span class="o">.</span><span class="n">SHUTDOWN</span>
+ <span class="k">if</span> <span class="n">activate_dag_runs</span><span class="p">:</span>
+ <span class="n">execution_dates</span> <span class="o">=</span> <span class="p">{</span><span class="n">ti</span><span class="o">.</span><span class="n">execution_date</span> <span class="k">for</span> <span class="n">ti</span> <span class="ow">in</span> <span class="n">tis</span><span class="p">}</span>
+ <span class="n">dag_ids</span> <span class="o">=</span> <span class="p">{</span><span class="n">ti</span><span class="o">.</span><span class="n">dag_id</span> <span class="k">for</span> <span class="n">ti</span> <span class="ow">in</span> <span class="n">tis</span><span class="p">}</span>
+ <span class="n">drs</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">DagRun</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span>
+ <span class="n">DagRun</span><span class="o">.</span><span class="n">dag_id</span><span class="o">.</span><span class="n">in_</span><span class="p">(</span><span class="n">dag_ids</span><span class="p">),</span>
+ <span class="n">DagRun</span><span class="o">.</span><span class="n">execution_date</span><span class="o">.</span><span class="n">in_</span><span class="p">(</span><span class="n">execution_dates</span><span class="p">),</span>
+ <span class="p">)</span><span class="o">.</span><span class="n">all</span><span class="p">()</span>
+ <span class="k">for</span> <span class="n">dr</span> <span class="ow">in</span> <span class="n">drs</span><span class="p">:</span>
+ <span class="n">dr</span><span class="o">.</span><span class="n">state</span> <span class="o">=</span> <span class="n">State</span><span class="o">.</span><span class="n">RUNNING</span>
+ <span class="n">dr</span><span class="o">.</span><span class="n">start_date</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+
+
+<div class="viewcode-block" id="DagBag"><a class="viewcode-back" href="../../code.html#airflow.models.DagBag">[docs]</a><span class="k">class</span> <span class="nc">DagBag</span><span class="p">(</span><span class="n">LoggingMixin</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> A dagbag is a collection of dags, parsed out of a folder tree and has high</span>
+<span class="sd"> level configuration settings, like what database to use as a backend and</span>
+<span class="sd"> what executor to use to fire off tasks. This makes it easier to run</span>
+<span class="sd"> distinct environments for say production and development, tests, or for</span>
+<span class="sd"> different teams or security profiles. What would have been system level</span>
+<span class="sd"> settings are now dagbag level so that one system can run multiple,</span>
+<span class="sd"> independent settings sets.</span>
+
+<span class="sd"> :param dag_folder: the folder to scan to find DAGs</span>
+<span class="sd"> :type dag_folder: str</span>
+<span class="sd"> :param executor: the executor to use when executing task instances</span>
+<span class="sd"> in this DagBag</span>
+<span class="sd"> :param include_examples: whether to include the examples that ship</span>
+<span class="sd"> with airflow or not</span>
+<span class="sd"> :type include_examples: bool</span>
+<span class="sd"> :param sync_to_db: whether to sync the properties of the DAGs to</span>
+<span class="sd"> the metadata DB while finding them, typically should be done</span>
+<span class="sd"> by the scheduler job only</span>
+<span class="sd"> :type sync_to_db: bool</span>
+<span class="sd"> """</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">dag_folder</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">executor</span><span class="o">=</span><span class="n">DEFAULT_EXECUTOR</span><span class="p">,</span>
+ <span class="n">include_examples</span><span class="o">=</span><span class="n">configuration</span><span class="o">.</span><span class="n">getboolean</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s1">'LOAD_EXAMPLES'</span><span class="p">),</span>
+ <span class="n">sync_to_db</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
+
+ <span class="n">dag_folder</span> <span class="o">=</span> <span class="n">dag_folder</span> <span class="ow">or</span> <span class="n">DAGS_FOLDER</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Filling up the DagBag from {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">dag_folder</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dag_folder</span> <span class="o">=</span> <span class="n">dag_folder</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dags</span> <span class="o">=</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sync_to_db</span> <span class="o">=</span> <span class="n">sync_to_db</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">file_last_changed</span> <span class="o">=</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">executor</span> <span class="o">=</span> <span class="n">executor</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">import_errors</span> <span class="o">=</span> <span class="p">{}</span>
+ <span class="k">if</span> <span class="n">include_examples</span><span class="p">:</span>
+ <span class="n">example_dag_folder</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span>
+ <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">__file__</span><span class="p">),</span>
+ <span class="s1">'example_dags'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">collect_dags</span><span class="p">(</span><span class="n">example_dag_folder</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">collect_dags</span><span class="p">(</span><span class="n">dag_folder</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">sync_to_db</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">deactivate_inactive_dags</span><span class="p">()</span>
+
+<div class="viewcode-block" id="DagBag.size"><a class="viewcode-back" href="../../code.html#airflow.models.DagBag.size">[docs]</a> <span class="k">def</span> <span class="nf">size</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> :return: the amount of dags contained in this dagbag</span>
+<span class="sd"> """</span>
+ <span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">dags</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="DagBag.get_dag"><a class="viewcode-back" href="../../code.html#airflow.models.DagBag.get_dag">[docs]</a> <span class="k">def</span> <span class="nf">get_dag</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dag_id</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Gets the DAG out of the dictionary, and refreshes it if expired</span>
+<span class="sd"> """</span>
+ <span class="c1"># If asking for a known subdag, we want to refresh the parent</span>
+ <span class="n">root_dag_id</span> <span class="o">=</span> <span class="n">dag_id</span>
+ <span class="k">if</span> <span class="n">dag_id</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">dags</span><span class="p">:</span>
+ <span class="n">dag</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">dags</span><span class="p">[</span><span class="n">dag_id</span><span class="p">]</span>
+ <span class="k">if</span> <span class="n">dag</span><span class="o">.</span><span class="n">is_subdag</span><span class="p">:</span>
+ <span class="n">root_dag_id</span> <span class="o">=</span> <span class="n">dag</span><span class="o">.</span><span class="n">parent_dag</span><span class="o">.</span><span class="n">dag_id</span>
+
+ <span class="c1"># If the root_dag_id is absent or expired</span>
+ <span class="n">orm_dag</span> <span class="o">=</span> <span class="n">DagModel</span><span class="o">.</span><span class="n">get_current</span><span class="p">(</span><span class="n">root_dag_id</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">orm_dag</span> <span class="ow">and</span> <span class="p">(</span>
+ <span class="n">root_dag_id</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">dags</span> <span class="ow">or</span>
+ <span class="p">(</span>
+ <span class="n">orm_dag</span><span class="o">.</span><span class="n">last_expired</span> <span class="ow">and</span>
+ <span class="n">dag</span><span class="o">.</span><span class="n">last_loaded</span> <span class="o"><</span> <span class="n">orm_dag</span><span class="o">.</span><span class="n">last_expired</span>
+ <span class="p">)</span>
+ <span class="p">):</span>
+ <span class="c1"># Reprocessing source file</span>
+ <span class="n">found_dags</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">process_file</span><span class="p">(</span>
+ <span class="n">filepath</span><span class="o">=</span><span class="n">orm_dag</span><span class="o">.</span><span class="n">fileloc</span><span class="p">,</span> <span class="n">only_if_updated</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="n">found_dags</span> <span class="ow">and</span> <span class="n">dag_id</span> <span class="ow">in</span> <span class="p">[</span><span class="n">dag</span><span class="o">.</span><span class="n">dag_id</span> <span class="k">for</span> <span class="n">dag</span> <span class="ow">in</span> <span class="n">found_dags</span><span class="p">]:</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">dags</span><span class="p">[</span><span class="n">dag_id</span><span class="p">]</span>
+ <span class="k">elif</span> <span class="n">dag_id</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">dags</span><span class="p">:</span>
+ <span class="k">del</span> <span class="bp">self</span><span class="o">.</span><span class="n">dags</span><span class="p">[</span><span class="n">dag_id</span><span class="p">]</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">dags</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">dag_id</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="DagBag.process_file"><a class="viewcode-back" href="../../code.html#airflow.models.DagBag.process_file">[docs]</a> <span class="k">def</span> <span class="nf">process_file</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">filepath</span><span class="p">,</span> <span class="n">only_if_updated</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">safe_mode</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Given a path to a python module or zip file, this method imports</span>
+<span class="sd"> the module and look for dag objects within it.</span>
+<span class="sd"> """</span>
+ <span class="n">found_dags</span> <span class="o">=</span> <span class="p">[]</span>
+
+ <span class="c1"># todo: raise exception?</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">filepath</span><span class="p">):</span>
+ <span class="k">return</span> <span class="n">found_dags</span>
+
+ <span class="k">try</span><span class="p">:</span>
+ <span class="c1"># This failed before in what may have been a git sync</span>
+ <span class="c1"># race condition</span>
+ <span class="n">dttm</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">fromtimestamp</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">getmtime</span><span class="p">(</span><span class="n">filepath</span><span class="p">))</span>
+ <span class="k">if</span> <span class="n">only_if_updated</span> \
+ <span class="ow">and</span> <span class="n">filepath</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">file_last_changed</span> \
+ <span class="ow">and</span> <span class="n">dttm</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">file_last_changed</span><span class="p">[</span><span class="n">filepath</span><span class="p">]:</span>
+ <span class="k">return</span> <span class="n">found_dags</span>
+
+ <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">exception</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">found_dags</span>
+
+ <span class="n">mods</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">zipfile</span><span class="o">.</span><span class="n">is_zipfile</span><span class="p">(</span><span class="n">filepath</span><span class="p">):</span>
+ <span class="k">if</span> <span class="n">safe_mode</span> <span class="ow">and</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">filepath</span><span class="p">):</span>
+ <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filepath</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
+ <span class="n">content</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">([</span><span class="n">s</span> <span class="ow">in</span> <span class="n">content</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="p">(</span><span class="n">b</span><span class="s1">'DAG'</span><span class="p">,</span> <span class="n">b</span><span class="s1">'airflow'</span><span class="p">)]):</span>
+ <span class="k">return</span> <span class="n">found_dags</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Importing {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">filepath</span><span class="p">))</span>
+ <span class="n">org_mod_name</span><span class="p">,</span> <span class="n">file_ext</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">splitext</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">filepath</span><span class="p">)[</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
+ <span class="n">mod_name</span> <span class="o">=</span> <span class="s1">'unusual_prefix_'</span> <span class="o">+</span> <span class="n">org_mod_name</span>
+
+ <span class="k">if</span> <span class="n">mod_name</span> <span class="ow">in</span> <span class="n">sys</span><span class="o">.</span><span class="n">modules</span><span class="p">:</span>
+ <span class="k">del</span> <span class="n">sys</span><span class="o">.</span><span class="n">modules</span><span class="p">[</span><span class="n">mod_name</span><span class="p">]</span>
+
+ <span class="k">with</span> <span class="n">timeout</span><span class="p">(</span><span class="n">configuration</span><span class="o">.</span><span class="n">getint</span><span class="p">(</span><span class="s1">'core'</span><span class="p">,</span> <span class="s2">"DAGBAG_IMPORT_TIMEOUT"</span><span class="p">)):</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">m</span> <span class="o">=</span> <span class="n">imp</span><span class="o">.</span><span class="n">load_source</span><span class="p">(</span><span class="n">mod_name</span><span class="p">,</span> <span class="n">filepath</span><span class="p">)</span>
+ <span class="n">mods</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
+ <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">exception</span><span class="p">(</span><span class="s2">"Failed to import: "</span> <span class="o">+</span> <span class="n">filepath</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">import_errors</span><span class="p">[</span><span class="n">filepath</span><span class="p">]</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">file_last_changed</span><span class="p">[</span><span class="n">filepath</span><span class="p">]</span> <span class="o">=</span> <span class="n">dttm</span>
+
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">zip_file</span> <span class="o">=</span> <span class="n">zipfile</span><span class="o">.</span><span class="n">ZipFile</span><span class="p">(</span><span class="n">filepath</span><span class="p">)</span>
+ <span class="k">for</span> <span class="n">mod</span> <span class="ow">in</span> <span class="n">zip_file</span><span class="o">.</span><span class="n">infolist</span><span class="p">():</span>
+ <span class="n">head</span><span class="p">,</span> <span class="n">tail</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">mod</span><span class="o">.</span><span class="n">filename</span><span class="p">)</span>
+ <span class="n">mod_name</span><span class="p">,</span> <span class="n">ext</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">splitext</span><span class="p">(</span><span class="n">mod</span><span class="o">.</span><span class="n">filename</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">head</span> <span class="ow">and</span> <span class="p">(</span><span class="n">ext</span> <span class="o">==</span> <span class="s1">'.py'</span> <span class="ow">or</span> <span class="n">ext</span> <span class="o">==</span> <span class="s1">'.pyc'</span><span class="p">):</span>
+ <span class="k">if</span> <span class="n">mod_name</span> <span class="o">==</span> <span class="s1">'__init__'</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"Found __init__.{0} at root of {1}"</span><span class="o">.</span>
+ <span class="n">format</span><span class="p">(</span><span class="n">ext</span><span class="p">,</span> <span class="n">filepath</span><span class="p">))</span>
+
+ <span class="k">if</span> <span class="n">safe_mode</span><span class="p">:</span>
+ <span class="k">with</span> <span class="n">zip_file</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">mod</span><span class="o">.</span><span class="n">filename</span><span class="p">)</span> <span class="k">as</span> <span class="n">zf</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Reading {} from {}"</span><span class="o">.</span>
+ <span class="n">format</span><span class="p">(</span><span class="n">mod</span><span class="o">.</span><span class="n">filename</span><span class="p">,</span> <span class="n">filepath</span><span class="p">))</span>
+ <span class="n">content</span> <span class="o">=</span> <span class="n">zf</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">([</span><span class="n">s</span> <span class="ow">in</span> <span class="n">content</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="p">(</span><span class="n">b</span><span class="s1">'DAG'</span><span class="p">,</span> <span class="n">b</span><span class="s1">'airflow'</span><span class="p">)]):</span>
+ <span class="c1"># todo: create ignore list</span>
+ <span class="k">return</span> <span class="n">found_dags</span>
+
+ <span class="k">if</span> <span class="n">mod_name</span> <span class="ow">in</span> <span class="n">sys</span><span class="o">.</span><span class="n">modules</span><span class="p">:</span>
+ <span class="k">del</span> <span class="n">sys</span><span class="o">.</span><span class="n">modules</span><span class="p">[</span><span class="n">mod_name</span><span class="p">]</span>
+
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">sys</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="n">filepath</span><span class="p">)</span>
+ <span class="n">m</span> <span class="o">=</span> <span class="n">importlib</span><span class="o">.</span><span class="n">import_module</span><span class="p">(</span><span class="n">mod_name</span><span class="p">)</span>
+ <span class="n">mods</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">m</span><span class="p">)</span>
+ <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">exception</span><span class="p">(</span><span class="s2">"Failed to import: "</span> <span class="o">+</span> <span class="n">filepath</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">import_errors</span><span class="p">[</span><span class="n">filepath</span><span class="p">]</span> <span class="o">=</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">file_last_changed</span><span class="p">[</span><span class="n">filepath</span><span class="p">]</span> <span class="o">=</span> <span class="n">dttm</span>
+
+ <span class="k">for</span> <span class="n">m</span> <span class="ow">in</span> <span class="n">mods</span><span class="p">:</span>
+ <span class="k">for</span> <span class="n">dag</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="n">m</span><span class="o">.</span><span class="n">__dict__</span><span class="o">.</span><span class="n">values</span><span class="p">()):</span>
+ <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">dag</span><span class="p">,</span> <span class="n">DAG</span><span class="p">):</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">dag</span><span class="o">.</span><span class="n">full_filepath</span><span class="p">:</span>
+ <span class="n">dag</span><span class="o">.</span><span class="n">full_filepath</span> <span class="o">=</span> <span class="n">filepath</span>
+ <span class="n">dag</span><span class="o">.</span><span class="n">is_subdag</span> <span class="o">=</span> <span class="bp">False</span>
+ <span class="n">dag</span><span class="o">.</span><span class="n">module_name</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">__name__</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bag_dag</span><span class="p">(</span><span class="n">dag</span><span class="p">,</span> <span class="n">parent_dag</span><span class="o">=</span><span class="n">dag</span><span class="p">,</span> <span class="n">root_dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span>
+ <span class="n">found_dags</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">dag</span><span class="p">)</span>
+ <span class="n">found_dags</span> <span class="o">+=</span> <span class="n">dag</span><span class="o">.</span><span class="n">subdags</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">file_last_changed</span><span class="p">[</span><span class="n">filepath</span><span class="p">]</span> <span class="o">=</span> <span class="n">dttm</span>
+ <span class="k">return</span> <span class="n">found_dags</span></div>
+
+ <span class="nd">@provide_session</span>
+<div class="viewcode-block" id="DagBag.kill_zombies"><a class="viewcode-back" href="../../code.html#airflow.models.DagBag.kill_zombies">[docs]</a> <span class="k">def</span> <span class="nf">kill_zombies</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">session</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Fails tasks that haven't had a heartbeat in too long</span>
+<span class="sd"> """</span>
+ <span class="kn">from</span> <span class="nn">airflow.jobs</span> <span class="kn">import</span> <span class="n">LocalTaskJob</span> <span class="k">as</span> <span class="n">LJ</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Finding 'running' jobs without a recent heartbeat"</span><span class="p">)</span>
+ <span class="n">TI</span> <span class="o">=</span> <span class="n">TaskInstance</span>
+ <span class="n">secs</span> <span class="o">=</span> <span class="p">(</span>
+ <span class="n">configuration</span><span class="o">.</span><span class="n">getint</span><span class="p">(</span><span class="s1">'scheduler'</span><span class="p">,</span> <span class="s1">'job_heartbeat_sec'</span><span class="p">)</span> <span class="o">*</span> <span class="mi">3</span><span class="p">)</span> <span class="o">+</span> <span class="mi">120</span>
+ <span class="n">limit_dttm</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">-</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">seconds</span><span class="o">=</span><span class="n">secs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s2">"Failing jobs without heartbeat after {}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">limit_dttm</span><span class="p">))</span>
+
+ <span class="n">tis</span> <span class="o">=</span> <span class="p">(</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">TI</span><span class="p">)</span>
+ <span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">LJ</span><span class="p">,</span> <span class="n">TI</span><span class="o">.</span><span class="n">job_id</span> <span class="o">==</span> <span class="n">LJ</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
+ <span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">TI</span><span class="o">.</span><span class="n">state</span> <span class="o">==</span> <span class="n">State</span><span class="o">.</span><span class="n">RUNNING</span><span class="p">)</span>
+ <span class="o">.</span><span class="n">filter</span><span class="p">(</span>
+ <span class="n">or_</span><span class="p">(</span>
+ <span class="n">LJ</span><span class="o">.</span><span class="n">state</span> <span class="o">!=</span> <span class="n">State</span><span class="o">.</span><span class="n">RUNNING</span><span class="p">,</span>
+ <span class="n">LJ</span><span class="o">.</span><span class="n">latest_heartbeat</span> <span class="o"><</span> <span class="n">limit_dttm</span><span class="p">,</span>
+ <span class="p">))</span>
+ <span class="o">.</span><span class="n">all</span><span class="p">()</span>
+ <span class="p">)</span>
+
+ <span class="k">for</span> <span class="n">ti</span> <span class="ow">in</span> <span class="n">tis</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">ti</span> <span class="ow">and</span> <span class="n">ti</span><span class="o">.</span><span class="n">dag_id</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">dags</span><span class="p">:</span>
+ <span class="n">dag</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">dags</span><span class="p">[</span><span class="n">ti</span><span class="o">.</span><span class="n">dag_id</span><span class="p">]</span>
+ <span class="k">if</span> <span class="n">ti</span><span class="o">.</span><span class="n">task_id</span> <span class="ow">in</span> <span class="n">dag</span><span class="o">.</span><span class="n">task_ids</span><span class="p">:</span>
+ <span class="n">task</span> <span class="o">=</span> <span class="n">dag</span><span class="o">.</span><span class="n">get_task</span><span class="p">(</span><span class="n">ti</span><span class="o">.</span><span class="n">task_id</span><span class="p">)</span>
+ <span class="n">ti</span><span class="o">.</span><span class="n">task</span> <span class="o">=</span> <span class="n">task</span>
+ <span class="n">ti</span><span class="o">.</span><span class="n">handle_failure</span><span class="p">(</span><span class="s2">"{} killed as zombie"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">ti</span><span class="p">))</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Marked zombie job {} as failed'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">ti</span><span class="p">))</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span></div>
+
+<div class="viewcode-block" id="DagBag.bag_dag"><a class="viewcode-back" href="../../code.html#airflow.models.DagBag.bag_dag">[docs]</a> <span class="k">def</span> <span class="nf">bag_dag</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">dag</span><span class="p">,</span> <span class="n">parent_dag</span><span class="p">,</span> <span class="n">root_dag</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Adds the DAG into the bag, recurses into sub dags.</span>
+<span class="sd"> """</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dags</span><span class="p">[</span><span class="n">dag</span><span class="o">.</span><span class="n">dag_id</span><span class="p">]</span> <span class="o">=</span> <span class="n">dag</span>
+ <span class="n">dag</span><span class="o">.</span><span class="n">resolve_template_files</span><span class="p">()</span>
+ <span class="n">dag</span><span class="o">.</span><span class="n">last_loaded</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+
+ <span class="k">for</span> <span class="n">task</span> <span class="ow">in</span> <span class="n">dag</span><span class="o">.</span><span class="n">tasks</span><span class="p">:</span>
+ <span class="n">settings</span><span class="o">.</span><span class="n">policy</span><span class="p">(</span><span class="n">task</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">sync_to_db</span><span class="p">:</span>
+ <span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
+ <span class="n">orm_dag</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span>
+ <span class="n">DagModel</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">DagModel</span><span class="o">.</span><span class="n">dag_id</span> <span class="o">==</span> <span class="n">dag</span><span class="o">.</span><span class="n">dag_id</span><span class="p">)</span><span class="o">.</span><span class="n">first</span><span class="p">()</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">orm_dag</span><span class="p">:</span>
+ <span class="n">orm_dag</span> <span class="o">=</span> <span class="n">DagModel</span><span class="p">(</span><span class="n">dag_id</span><span class="o">=</span><span class="n">dag</span><span class="o">.</span><span class="n">dag_id</span><span class="p">)</span>
+ <span class="n">orm_dag</span><span class="o">.</span><span class="n">fileloc</span> <span class="o">=</span> <span class="n">root_dag</span><span class="o">.</span><span class="n">full_filepath</span>
+ <span class="n">orm_dag</span><span class="o">.</span><span class="n">is_subdag</span> <span class="o">=</span> <span class="n">dag</span><span class="o">.</span><span class="n">is_subdag</span>
+ <span class="n">orm_dag</span><span class="o">.</span><span class="n">owners</span> <span class="o">=</span> <span class="n">root_dag</span><span class="o">.</span><span class="n">owner</span>
+ <span class="n">orm_dag</span><span class="o">.</span><span class="n">is_active</span> <span class="o">=</span> <span class="bp">True</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span><span class="n">orm_dag</span><span class="p">)</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+
+ <span class="k">for</span> <span class="n">subdag</span> <span class="ow">in</span> <span class="n">dag</span><span class="o">.</span><span class="n">subdags</span><span class="p">:</span>
+ <span class="n">subdag</span><span class="o">.</span><span class="n">full_filepath</span> <span class="o">=</span> <span class="n">dag</span><span class="o">.</span><span class="n">full_filepath</span>
+ <span class="n">subdag</span><span class="o">.</span><span class="n">parent_dag</span> <span class="o">=</span> <span class="n">dag</span>
+ <span class="n">subdag</span><span class="o">.</span><span class="n">fileloc</span> <span class="o">=</span> <span class="n">root_dag</span><span class="o">.</span><span class="n">full_filepath</span>
+ <span class="n">subdag</span><span class="o">.</span><span class="n">is_subdag</span> <span class="o">=</span> <span class="bp">True</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bag_dag</span><span class="p">(</span><span class="n">subdag</span><span class="p">,</span> <span class="n">parent_dag</span><span class="o">=</span><span class="n">dag</span><span class="p">,</span> <span class="n">root_dag</span><span class="o">=</span><span class="n">root_dag</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'Loaded DAG {dag}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span></div>
+
+<div class="viewcode-block" id="DagBag.collect_dags"><a class="viewcode-back" href="../../code.html#airflow.models.DagBag.collect_dags">[docs]</a> <span class="k">def</span> <span class="nf">collect_dags</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">dag_folder</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">only_if_updated</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Given a file path or a folder, this method looks for python modules,</span>
+<span class="sd"> imports them and adds them to the dagbag collection.</span>
+
+<span class="sd"> Note that if a .airflowignore file is found while processing,</span>
+<span class="sd"> the directory, it will behaves much like a .gitignore does,</span>
+<span class="sd"> ignoring files that match any of the regex patterns specified</span>
+<span class="sd"> in the file.</span>
+<span class="sd"> """</span>
+ <span class="n">start_dttm</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+ <span class="n">dag_folder</span> <span class="o">=</span> <span class="n">dag_folder</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">dag_folder</span>
+
+ <span class="c1"># Used to store stats around DagBag processing</span>
+ <span class="n">stats</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="n">FileLoadStat</span> <span class="o">=</span> <span class="n">namedtuple</span><span class="p">(</span>
+ <span class="s1">'FileLoadStat'</span><span class="p">,</span> <span class="s2">"file duration dag_num task_num dags"</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">dag_folder</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">process_file</span><span class="p">(</span><span class="n">dag_folder</span><span class="p">,</span> <span class="n">only_if_updated</span><span class="o">=</span><span class="n">only_if_updated</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isdir</span><span class="p">(</span><span class="n">dag_folder</span><span class="p">):</span>
+ <span class="n">patterns</span> <span class="o">=</span> <span class="p">[]</span>
+ <span class="k">for</span> <span class="n">root</span><span class="p">,</span> <span class="n">dirs</span><span class="p">,</span> <span class="n">files</span> <span class="ow">in</span> <span class="n">os</span><span class="o">.</span><span class="n">walk</span><span class="p">(</span><span class="n">dag_folder</span><span class="p">,</span> <span class="n">followlinks</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span>
+ <span class="n">ignore_file</span> <span class="o">=</span> <span class="p">[</span><span class="n">f</span> <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">files</span> <span class="k">if</span> <span class="n">f</span> <span class="o">==</span> <span class="s1">'.airflowignore'</span><span class="p">]</span>
+ <span class="k">if</span> <span class="n">ignore_file</span><span class="p">:</span>
+ <span class="n">f</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">ignore_file</span><span class="p">[</span><span class="mi">0</span><span class="p">]),</span> <span class="s1">'r'</span><span class="p">)</span>
+ <span class="n">patterns</span> <span class="o">+=</span> <span class="p">[</span><span class="n">p</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span> <span class="k">if</span> <span class="n">p</span><span class="p">]</span>
+ <span class="n">f</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">files</span><span class="p">:</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">filepath</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">root</span><span class="p">,</span> <span class="n">f</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">filepath</span><span class="p">):</span>
+ <span class="k">continue</span>
+ <span class="n">mod_name</span><span class="p">,</span> <span class="n">file_ext</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">splitext</span><span class="p">(</span>
+ <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">filepath</span><span class="p">)[</span><span class="o">-</span><span class="mi">1</span><span class="p">])</span>
+ <span class="k">if</span> <span class="n">file_ext</span> <span class="o">!=</span> <span class="s1">'.py'</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">zipfile</span><span class="o">.</span><span class="n">is_zipfile</span><span class="p">(</span><span class="n">filepath</span><span class="p">):</span>
+ <span class="k">continue</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="nb">any</span><span class="p">(</span>
+ <span class="p">[</span><span class="n">re</span><span class="o">.</span><span class="n">findall</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">filepath</span><span class="p">)</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">patterns</span><span class="p">]):</span>
+ <span class="n">ts</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+ <span class="n">found_dags</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">process_file</span><span class="p">(</span>
+ <span class="n">filepath</span><span class="p">,</span> <span class="n">only_if_updated</span><span class="o">=</span><span class="n">only_if_updated</span><span class="p">)</span>
+
+ <span class="n">td</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">-</span> <span class="n">ts</span>
+ <span class="n">td</span> <span class="o">=</span> <span class="n">td</span><span class="o">.</span><span class="n">total_seconds</span><span class="p">()</span> <span class="o">+</span> <span class="p">(</span>
+ <span class="nb">float</span><span class="p">(</span><span class="n">td</span><span class="o">.</span><span class="n">microseconds</span><span class="p">)</span> <span class="o">/</span> <span class="mi">1000000</span><span class="p">)</span>
+ <span class="n">stats</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">FileLoadStat</span><span class="p">(</span>
+ <span class="n">filepath</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">dag_folder</span><span class="p">,</span> <span class="s1">''</span><span class="p">),</span>
+ <span class="n">td</span><span class="p">,</span>
+ <span class="nb">len</span><span class="p">(</span><span class="n">found_dags</span><span class="p">),</span>
+ <span class="nb">sum</span><span class="p">([</span><span class="nb">len</span><span class="p">(</span><span class="n">dag</span><span class="o">.</span><span class="n">tasks</span><span class="p">)</span> <span class="k">for</span> <span class="n">dag</span> <span class="ow">in</span> <span class="n">found_dags</span><span class="p">]),</span>
+ <span class="nb">str</span><span class="p">([</span><span class="n">dag</span><span class="o">.</span><span class="n">dag_id</span> <span class="k">for</span> <span class="n">dag</span> <span class="ow">in</span> <span class="n">found_dags</span><span class="p">]),</span>
+ <span class="p">))</span>
+ <span class="k">except</span> <span class="ne">Exception</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
+ <span class="n">Stats</span><span class="o">.</span><span class="n">gauge</span><span class="p">(</span>
+ <span class="s1">'collect_dags'</span><span class="p">,</span> <span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">-</span> <span class="n">start_dttm</span><span class="p">)</span><span class="o">.</span><span class="n">total_seconds</span><span class="p">(),</span> <span class="mi">1</span><span class="p">)</span>
+ <span class="n">Stats</span><span class="o">.</span><span class="n">gauge</span><span class="p">(</span>
+ <span class="s1">'dagbag_size'</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">dags</span><span class="p">),</span> <span class="mi">1</span><span class="p">)</span>
+ <span class="n">Stats</span><span class="o">.</span><span class="n">gauge</span><span class="p">(</span>
+ <span class="s1">'dagbag_import_errors'</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">import_errors</span><span class="p">),</span> <span class="mi">1</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">dagbag_stats</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">(</span>
+ <span class="n">stats</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="o">.</span><span class="n">duration</span><span class="p">,</span> <span class="n">reverse</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="DagBag.dagbag_report"><a class="viewcode-back" href="../../code.html#airflow.models.DagBag.dagbag_report">[docs]</a> <span class="k">def</span> <span class="nf">dagbag_report</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""Prints a report around DagBag loading stats"""</span>
+ <span class="n">report</span> <span class="o">=</span> <span class="n">textwrap</span><span class="o">.</span><span class="n">dedent</span><span class="p">(</span><span class="s2">"""</span><span class="se">\n</span><span class="s2"></span>
+<span class="s2"> -------------------------------------------------------------------</span>
+<span class="s2"> DagBag loading stats for {dag_folder}</span>
+<span class="s2"> -------------------------------------------------------------------</span>
+<span class="s2"> Number of DAGs: {dag_num}</span>
+<span class="s2"> Total task number: {task_num}</span>
+<span class="s2"> DagBag parsing time: {duration}</span>
+<span class="s2"> {table}</span>
+<span class="s2"> """</span><span class="p">)</span>
+ <span class="n">stats</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">dagbag_stats</span>
+ <span class="k">return</span> <span class="n">report</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">dag_folder</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">dag_folder</span><span class="p">,</span>
+ <span class="n">duration</span><span class="o">=</span><span class="nb">sum</span><span class="p">([</span><span class="n">o</span><span class="o">.</span><span class="n">duration</span> <span class="k">for</span> <span class="n">o</span> <span class="ow">in</span> <span class="n">stats</span><span class="p">]),</span>
+ <span class="n">dag_num</span><span class="o">=</span><span class="nb">sum</span><span class="p">([</span><span class="n">o</span><span class="o">.</span><span class="n">dag_num</span> <span class="k">for</span> <span class="n">o</span> <span class="ow">in</span> <span class="n">stats</span><span class="p">]),</span>
+ <span class="n">task_num</span><span class="o">=</span><span class="nb">sum</span><span class="p">([</span><span class="n">o</span><span class="o">.</span><span class="n">dag_num</span> <span class="k">for</span> <span class="n">o</span> <span class="ow">in</span> <span class="n">stats</span><span class="p">]),</span>
+ <span class="n">table</span><span class="o">=</span><span class="n">pprinttable</span><span class="p">(</span><span class="n">stats</span><span class="p">),</span>
+ <span class="p">)</span></div>
+
+ <span class="k">def</span> <span class="nf">deactivate_inactive_dags</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">active_dag_ids</span> <span class="o">=</span> <span class="p">[</span><span class="n">dag</span><span class="o">.</span><span class="n">dag_id</span> <span class="k">for</span> <span class="n">dag</span> <span class="ow">in</span> <span class="nb">list</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">dags</span><span class="o">.</span><span class="n">values</span><span class="p">())]</span>
+ <span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
+ <span class="k">for</span> <span class="n">dag</span> <span class="ow">in</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span>
+ <span class="n">DagModel</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="o">~</span><span class="n">DagModel</span><span class="o">.</span><span class="n">dag_id</span><span class="o">.</span><span class="n">in_</span><span class="p">(</span><span class="n">active_dag_ids</span><span class="p">))</span><span class="o">.</span><span class="n">all</span><span class="p">():</span>
+ <span class="n">dag</span><span class="o">.</span><span class="n">is_active</span> <span class="o">=</span> <span class="bp">False</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">merge</span><span class="p">(</span><span class="n">dag</span><span class="p">)</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+
+ <span class="k">def</span> <span class="nf">paused_dags</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
+ <span class="n">dag_ids</span> <span class="o">=</span> <span class="p">[</span><span class="n">dp</span><span class="o">.</span><span class="n">dag_id</span> <span class="k">for</span> <span class="n">dp</span> <span class="ow">in</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">DagModel</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span>
+ <span class="n">DagModel</span><span class="o">.</span><span class="n">is_paused</span> <span class="o">==</span> <span class="bp">True</span><span class="p">)]</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">dag_ids</span></div>
+
+
+<span class="k">class</span> <span class="nc">User</span><span class="p">(</span><span class="n">Base</span><span class="p">):</span>
+ <span class="n">__tablename__</span> <span class="o">=</span> <span class="s2">"users"</span>
+
+ <span class="nb">id</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">Integer</span><span class="p">,</span> <span class="n">primary_key</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
+ <span class="n">username</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">String</span><span class="p">(</span><span class="n">ID_LEN</span><span class="p">),</span> <span class="n">unique</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
+ <span class="n">email</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">String</span><span class="p">(</span><span class="mi">500</span><span class="p">))</span>
+ <span class="n">superuser</span> <span class="o">=</span> <span class="bp">False</span>
+
+ <span class="k">def</span> <span class="nf">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">username</span>
+
+ <span class="k">def</span> <span class="nf">get_id</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="nb">str</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">id</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">is_superuser</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">superuser</span>
+
+
+<div class="viewcode-block" id="Connection"><a class="viewcode-back" href="../../code.html#airflow.models.Connection">[docs]</a><span class="k">class</span> <span class="nc">Connection</span><span class="p">(</span><span class="n">Base</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Placeholder to store information about different database instances</span>
+<span class="sd"> connection information. The idea here is that scripts use references to</span>
+<span class="sd"> database instances (conn_id) instead of hard coding hostname, logins and</span>
+<span class="sd"> passwords when using operators or hooks.</span>
+<span class="sd"> """</span>
+ <span class="n">__tablename__</span> <span class="o">=</span> <span class="s2">"connection"</span>
+
+ <span class="nb">id</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">Integer</span><span class="p">(),</span> <span class="n">primary_key</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
+ <span class="n">conn_id</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">String</span><span class="p">(</span><span class="n">ID_LEN</span><span class="p">))</span>
+ <span class="n">conn_type</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">String</span><span class="p">(</span><span class="mi">500</span><span class="p">))</span>
+ <span class="n">host</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">String</span><span class="p">(</span><span class="mi">500</span><span class="p">))</span>
+ <span class="n">schema</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">String</span><span class="p">(</span><span class="mi">500</span><span class="p">))</span>
+ <span class="n">login</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">String</span><span class="p">(</span><span class="mi">500</span><span class="p">))</span>
+ <span class="n">_password</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="s1">'password'</span><span class="p">,</span> <span class="n">String</span><span class="p">(</span><span class="mi">5000</span><span class="p">))</span>
+ <span class="n">port</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">Integer</span><span class="p">())</span>
+ <span class="n">is_encrypted</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">Boolean</span><span class="p">,</span> <span class="n">unique</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
+ <span class="n">is_extra_encrypted</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">Boolean</span><span class="p">,</span> <span class="n">unique</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
+ <span class="n">_extra</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="s1">'extra'</span><span class="p">,</span> <span class="n">String</span><span class="p">(</span><span class="mi">5000</span><span class="p">))</span>
+
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">conn_id</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">conn_type</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">host</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">login</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">password</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">schema</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">extra</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">uri</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span> <span class="o">=</span> <span class="n">conn_id</span>
+ <span class="k">if</span> <span class="n">uri</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">parse_from_uri</span><span class="p">(</span><span class="n">uri</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">conn_type</span> <span class="o">=</span> <span class="n">conn_type</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">host</span> <span class="o">=</span> <span class="n">host</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">login</span> <span class="o">=</span> <span class="n">login</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">password</span> <span class="o">=</span> <span class="n">password</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="n">schema</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">port</span> <span class="o">=</span> <span class="n">port</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">extra</span> <span class="o">=</span> <span class="n">extra</span>
+
+ <span class="k">def</span> <span class="nf">parse_from_uri</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">uri</span><span class="p">):</span>
+ <span class="n">temp_uri</span> <span class="o">=</span> <span class="n">urlparse</span><span class="p">(</span><span class="n">uri</span><span class="p">)</span>
+ <span class="n">hostname</span> <span class="o">=</span> <span class="n">temp_uri</span><span class="o">.</span><span class="n">hostname</span> <span class="ow">or</span> <span class="s1">''</span>
+ <span class="k">if</span> <span class="s1">'</span><span class="si">%2f</span><span class="s1">'</span> <span class="ow">in</span> <span class="n">hostname</span><span class="p">:</span>
+ <span class="n">hostname</span> <span class="o">=</span> <span class="n">hostname</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'</span><span class="si">%2f</span><span class="s1">'</span><span class="p">,</span> <span class="s1">'/'</span><span class="p">)</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'</span><span class="si">%2F</span><span class="s1">'</span><span class="p">,</span> <span class="s1">'/'</span><span class="p">)</span>
+ <span class="n">conn_type</span> <span class="o">=</span> <span class="n">temp_uri</span><span class="o">.</span><span class="n">scheme</span>
+ <span class="k">if</span> <span class="n">conn_type</span> <span class="o">==</span> <span class="s1">'postgresql'</span><span class="p">:</span>
+ <span class="n">conn_type</span> <span class="o">=</span> <span class="s1">'postgres'</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">conn_type</span> <span class="o">=</span> <span class="n">conn_type</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">host</span> <span class="o">=</span> <span class="n">hostname</spa
<TRUNCATED>
[08/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/underscore.js
----------------------------------------------------------------------
diff --git a/_static/underscore.js b/_static/underscore.js
new file mode 100644
index 0000000..5b55f32
--- /dev/null
+++ b/_static/underscore.js
@@ -0,0 +1,31 @@
+// Underscore.js 1.3.1
+// (c) 2009-2012 Jeremy Ashkenas, DocumentCloud Inc.
+// Underscore is freely distributable under the MIT license.
+// Portions of Underscore are inspired or borrowed from Prototype,
+// Oliver Steele's Functional, and John Resig's Micro-Templating.
+// For all details and documentation:
+// http://documentcloud.github.com/underscore
+(function(){function q(a,c,d){if(a===c)return a!==0||1/a==1/c;if(a==null||c==null)return a===c;if(a._chain)a=a._wrapped;if(c._chain)c=c._wrapped;if(a.isEqual&&b.isFunction(a.isEqual))return a.isEqual(c);if(c.isEqual&&b.isFunction(c.isEqual))return c.isEqual(a);var e=l.call(a);if(e!=l.call(c))return false;switch(e){case "[object String]":return a==String(c);case "[object Number]":return a!=+a?c!=+c:a==0?1/a==1/c:a==+c;case "[object Date]":case "[object Boolean]":return+a==+c;case "[object RegExp]":return a.source==
+c.source&&a.global==c.global&&a.multiline==c.multiline&&a.ignoreCase==c.ignoreCase}if(typeof a!="object"||typeof c!="object")return false;for(var f=d.length;f--;)if(d[f]==a)return true;d.push(a);var f=0,g=true;if(e=="[object Array]"){if(f=a.length,g=f==c.length)for(;f--;)if(!(g=f in a==f in c&&q(a[f],c[f],d)))break}else{if("constructor"in a!="constructor"in c||a.constructor!=c.constructor)return false;for(var h in a)if(b.has(a,h)&&(f++,!(g=b.has(c,h)&&q(a[h],c[h],d))))break;if(g){for(h in c)if(b.has(c,
+h)&&!f--)break;g=!f}}d.pop();return g}var r=this,G=r._,n={},k=Array.prototype,o=Object.prototype,i=k.slice,H=k.unshift,l=o.toString,I=o.hasOwnProperty,w=k.forEach,x=k.map,y=k.reduce,z=k.reduceRight,A=k.filter,B=k.every,C=k.some,p=k.indexOf,D=k.lastIndexOf,o=Array.isArray,J=Object.keys,s=Function.prototype.bind,b=function(a){return new m(a)};if(typeof exports!=="undefined"){if(typeof module!=="undefined"&&module.exports)exports=module.exports=b;exports._=b}else r._=b;b.VERSION="1.3.1";var j=b.each=
+b.forEach=function(a,c,d){if(a!=null)if(w&&a.forEach===w)a.forEach(c,d);else if(a.length===+a.length)for(var e=0,f=a.length;e<f;e++){if(e in a&&c.call(d,a[e],e,a)===n)break}else for(e in a)if(b.has(a,e)&&c.call(d,a[e],e,a)===n)break};b.map=b.collect=function(a,c,b){var e=[];if(a==null)return e;if(x&&a.map===x)return a.map(c,b);j(a,function(a,g,h){e[e.length]=c.call(b,a,g,h)});if(a.length===+a.length)e.length=a.length;return e};b.reduce=b.foldl=b.inject=function(a,c,d,e){var f=arguments.length>2;a==
+null&&(a=[]);if(y&&a.reduce===y)return e&&(c=b.bind(c,e)),f?a.reduce(c,d):a.reduce(c);j(a,function(a,b,i){f?d=c.call(e,d,a,b,i):(d=a,f=true)});if(!f)throw new TypeError("Reduce of empty array with no initial value");return d};b.reduceRight=b.foldr=function(a,c,d,e){var f=arguments.length>2;a==null&&(a=[]);if(z&&a.reduceRight===z)return e&&(c=b.bind(c,e)),f?a.reduceRight(c,d):a.reduceRight(c);var g=b.toArray(a).reverse();e&&!f&&(c=b.bind(c,e));return f?b.reduce(g,c,d,e):b.reduce(g,c)};b.find=b.detect=
+function(a,c,b){var e;E(a,function(a,g,h){if(c.call(b,a,g,h))return e=a,true});return e};b.filter=b.select=function(a,c,b){var e=[];if(a==null)return e;if(A&&a.filter===A)return a.filter(c,b);j(a,function(a,g,h){c.call(b,a,g,h)&&(e[e.length]=a)});return e};b.reject=function(a,c,b){var e=[];if(a==null)return e;j(a,function(a,g,h){c.call(b,a,g,h)||(e[e.length]=a)});return e};b.every=b.all=function(a,c,b){var e=true;if(a==null)return e;if(B&&a.every===B)return a.every(c,b);j(a,function(a,g,h){if(!(e=
+e&&c.call(b,a,g,h)))return n});return e};var E=b.some=b.any=function(a,c,d){c||(c=b.identity);var e=false;if(a==null)return e;if(C&&a.some===C)return a.some(c,d);j(a,function(a,b,h){if(e||(e=c.call(d,a,b,h)))return n});return!!e};b.include=b.contains=function(a,c){var b=false;if(a==null)return b;return p&&a.indexOf===p?a.indexOf(c)!=-1:b=E(a,function(a){return a===c})};b.invoke=function(a,c){var d=i.call(arguments,2);return b.map(a,function(a){return(b.isFunction(c)?c||a:a[c]).apply(a,d)})};b.pluck=
+function(a,c){return b.map(a,function(a){return a[c]})};b.max=function(a,c,d){if(!c&&b.isArray(a))return Math.max.apply(Math,a);if(!c&&b.isEmpty(a))return-Infinity;var e={computed:-Infinity};j(a,function(a,b,h){b=c?c.call(d,a,b,h):a;b>=e.computed&&(e={value:a,computed:b})});return e.value};b.min=function(a,c,d){if(!c&&b.isArray(a))return Math.min.apply(Math,a);if(!c&&b.isEmpty(a))return Infinity;var e={computed:Infinity};j(a,function(a,b,h){b=c?c.call(d,a,b,h):a;b<e.computed&&(e={value:a,computed:b})});
+return e.value};b.shuffle=function(a){var b=[],d;j(a,function(a,f){f==0?b[0]=a:(d=Math.floor(Math.random()*(f+1)),b[f]=b[d],b[d]=a)});return b};b.sortBy=function(a,c,d){return b.pluck(b.map(a,function(a,b,g){return{value:a,criteria:c.call(d,a,b,g)}}).sort(function(a,b){var c=a.criteria,d=b.criteria;return c<d?-1:c>d?1:0}),"value")};b.groupBy=function(a,c){var d={},e=b.isFunction(c)?c:function(a){return a[c]};j(a,function(a,b){var c=e(a,b);(d[c]||(d[c]=[])).push(a)});return d};b.sortedIndex=function(a,
+c,d){d||(d=b.identity);for(var e=0,f=a.length;e<f;){var g=e+f>>1;d(a[g])<d(c)?e=g+1:f=g}return e};b.toArray=function(a){return!a?[]:a.toArray?a.toArray():b.isArray(a)?i.call(a):b.isArguments(a)?i.call(a):b.values(a)};b.size=function(a){return b.toArray(a).length};b.first=b.head=function(a,b,d){return b!=null&&!d?i.call(a,0,b):a[0]};b.initial=function(a,b,d){return i.call(a,0,a.length-(b==null||d?1:b))};b.last=function(a,b,d){return b!=null&&!d?i.call(a,Math.max(a.length-b,0)):a[a.length-1]};b.rest=
+b.tail=function(a,b,d){return i.call(a,b==null||d?1:b)};b.compact=function(a){return b.filter(a,function(a){return!!a})};b.flatten=function(a,c){return b.reduce(a,function(a,e){if(b.isArray(e))return a.concat(c?e:b.flatten(e));a[a.length]=e;return a},[])};b.without=function(a){return b.difference(a,i.call(arguments,1))};b.uniq=b.unique=function(a,c,d){var d=d?b.map(a,d):a,e=[];b.reduce(d,function(d,g,h){if(0==h||(c===true?b.last(d)!=g:!b.include(d,g)))d[d.length]=g,e[e.length]=a[h];return d},[]);
+return e};b.union=function(){return b.uniq(b.flatten(arguments,true))};b.intersection=b.intersect=function(a){var c=i.call(arguments,1);return b.filter(b.uniq(a),function(a){return b.every(c,function(c){return b.indexOf(c,a)>=0})})};b.difference=function(a){var c=b.flatten(i.call(arguments,1));return b.filter(a,function(a){return!b.include(c,a)})};b.zip=function(){for(var a=i.call(arguments),c=b.max(b.pluck(a,"length")),d=Array(c),e=0;e<c;e++)d[e]=b.pluck(a,""+e);return d};b.indexOf=function(a,c,
+d){if(a==null)return-1;var e;if(d)return d=b.sortedIndex(a,c),a[d]===c?d:-1;if(p&&a.indexOf===p)return a.indexOf(c);for(d=0,e=a.length;d<e;d++)if(d in a&&a[d]===c)return d;return-1};b.lastIndexOf=function(a,b){if(a==null)return-1;if(D&&a.lastIndexOf===D)return a.lastIndexOf(b);for(var d=a.length;d--;)if(d in a&&a[d]===b)return d;return-1};b.range=function(a,b,d){arguments.length<=1&&(b=a||0,a=0);for(var d=arguments[2]||1,e=Math.max(Math.ceil((b-a)/d),0),f=0,g=Array(e);f<e;)g[f++]=a,a+=d;return g};
+var F=function(){};b.bind=function(a,c){var d,e;if(a.bind===s&&s)return s.apply(a,i.call(arguments,1));if(!b.isFunction(a))throw new TypeError;e=i.call(arguments,2);return d=function(){if(!(this instanceof d))return a.apply(c,e.concat(i.call(arguments)));F.prototype=a.prototype;var b=new F,g=a.apply(b,e.concat(i.call(arguments)));return Object(g)===g?g:b}};b.bindAll=function(a){var c=i.call(arguments,1);c.length==0&&(c=b.functions(a));j(c,function(c){a[c]=b.bind(a[c],a)});return a};b.memoize=function(a,
+c){var d={};c||(c=b.identity);return function(){var e=c.apply(this,arguments);return b.has(d,e)?d[e]:d[e]=a.apply(this,arguments)}};b.delay=function(a,b){var d=i.call(arguments,2);return setTimeout(function(){return a.apply(a,d)},b)};b.defer=function(a){return b.delay.apply(b,[a,1].concat(i.call(arguments,1)))};b.throttle=function(a,c){var d,e,f,g,h,i=b.debounce(function(){h=g=false},c);return function(){d=this;e=arguments;var b;f||(f=setTimeout(function(){f=null;h&&a.apply(d,e);i()},c));g?h=true:
+a.apply(d,e);i();g=true}};b.debounce=function(a,b){var d;return function(){var e=this,f=arguments;clearTimeout(d);d=setTimeout(function(){d=null;a.apply(e,f)},b)}};b.once=function(a){var b=false,d;return function(){if(b)return d;b=true;return d=a.apply(this,arguments)}};b.wrap=function(a,b){return function(){var d=[a].concat(i.call(arguments,0));return b.apply(this,d)}};b.compose=function(){var a=arguments;return function(){for(var b=arguments,d=a.length-1;d>=0;d--)b=[a[d].apply(this,b)];return b[0]}};
+b.after=function(a,b){return a<=0?b():function(){if(--a<1)return b.apply(this,arguments)}};b.keys=J||function(a){if(a!==Object(a))throw new TypeError("Invalid object");var c=[],d;for(d in a)b.has(a,d)&&(c[c.length]=d);return c};b.values=function(a){return b.map(a,b.identity)};b.functions=b.methods=function(a){var c=[],d;for(d in a)b.isFunction(a[d])&&c.push(d);return c.sort()};b.extend=function(a){j(i.call(arguments,1),function(b){for(var d in b)a[d]=b[d]});return a};b.defaults=function(a){j(i.call(arguments,
+1),function(b){for(var d in b)a[d]==null&&(a[d]=b[d])});return a};b.clone=function(a){return!b.isObject(a)?a:b.isArray(a)?a.slice():b.extend({},a)};b.tap=function(a,b){b(a);return a};b.isEqual=function(a,b){return q(a,b,[])};b.isEmpty=function(a){if(b.isArray(a)||b.isString(a))return a.length===0;for(var c in a)if(b.has(a,c))return false;return true};b.isElement=function(a){return!!(a&&a.nodeType==1)};b.isArray=o||function(a){return l.call(a)=="[object Array]"};b.isObject=function(a){return a===Object(a)};
+b.isArguments=function(a){return l.call(a)=="[object Arguments]"};if(!b.isArguments(arguments))b.isArguments=function(a){return!(!a||!b.has(a,"callee"))};b.isFunction=function(a){return l.call(a)=="[object Function]"};b.isString=function(a){return l.call(a)=="[object String]"};b.isNumber=function(a){return l.call(a)=="[object Number]"};b.isNaN=function(a){return a!==a};b.isBoolean=function(a){return a===true||a===false||l.call(a)=="[object Boolean]"};b.isDate=function(a){return l.call(a)=="[object Date]"};
+b.isRegExp=function(a){return l.call(a)=="[object RegExp]"};b.isNull=function(a){return a===null};b.isUndefined=function(a){return a===void 0};b.has=function(a,b){return I.call(a,b)};b.noConflict=function(){r._=G;return this};b.identity=function(a){return a};b.times=function(a,b,d){for(var e=0;e<a;e++)b.call(d,e)};b.escape=function(a){return(""+a).replace(/&/g,"&").replace(/</g,"<").replace(/>/g,">").replace(/"/g,""").replace(/'/g,"'").replace(/\//g,"/")};b.mixin=function(a){j(b.functions(a),
+function(c){K(c,b[c]=a[c])})};var L=0;b.uniqueId=function(a){var b=L++;return a?a+b:b};b.templateSettings={evaluate:/<%([\s\S]+?)%>/g,interpolate:/<%=([\s\S]+?)%>/g,escape:/<%-([\s\S]+?)%>/g};var t=/.^/,u=function(a){return a.replace(/\\\\/g,"\\").replace(/\\'/g,"'")};b.template=function(a,c){var d=b.templateSettings,d="var __p=[],print=function(){__p.push.apply(__p,arguments);};with(obj||{}){__p.push('"+a.replace(/\\/g,"\\\\").replace(/'/g,"\\'").replace(d.escape||t,function(a,b){return"',_.escape("+
+u(b)+"),'"}).replace(d.interpolate||t,function(a,b){return"',"+u(b)+",'"}).replace(d.evaluate||t,function(a,b){return"');"+u(b).replace(/[\r\n\t]/g," ")+";__p.push('"}).replace(/\r/g,"\\r").replace(/\n/g,"\\n").replace(/\t/g,"\\t")+"');}return __p.join('');",e=new Function("obj","_",d);return c?e(c,b):function(a){return e.call(this,a,b)}};b.chain=function(a){return b(a).chain()};var m=function(a){this._wrapped=a};b.prototype=m.prototype;var v=function(a,c){return c?b(a).chain():a},K=function(a,c){m.prototype[a]=
+function(){var a=i.call(arguments);H.call(a,this._wrapped);return v(c.apply(b,a),this._chain)}};b.mixin(b);j("pop,push,reverse,shift,sort,splice,unshift".split(","),function(a){var b=k[a];m.prototype[a]=function(){var d=this._wrapped;b.apply(d,arguments);var e=d.length;(a=="shift"||a=="splice")&&e===0&&delete d[0];return v(d,this._chain)}});j(["concat","join","slice"],function(a){var b=k[a];m.prototype[a]=function(){return v(b.apply(this._wrapped,arguments),this._chain)}});m.prototype.chain=function(){this._chain=
+true;return this};m.prototype.value=function(){return this._wrapped}}).call(this);
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/up-pressed.png
----------------------------------------------------------------------
diff --git a/_static/up-pressed.png b/_static/up-pressed.png
new file mode 100644
index 0000000..99e7210
Binary files /dev/null and b/_static/up-pressed.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/up.png
----------------------------------------------------------------------
diff --git a/_static/up.png b/_static/up.png
new file mode 100644
index 0000000..26de002
Binary files /dev/null and b/_static/up.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_static/websupport.js
----------------------------------------------------------------------
diff --git a/_static/websupport.js b/_static/websupport.js
new file mode 100644
index 0000000..98e7f40
--- /dev/null
+++ b/_static/websupport.js
@@ -0,0 +1,808 @@
+/*
+ * websupport.js
+ * ~~~~~~~~~~~~~
+ *
+ * sphinx.websupport utilities for all documentation.
+ *
+ * :copyright: Copyright 2007-2016 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+(function($) {
+ $.fn.autogrow = function() {
+ return this.each(function() {
+ var textarea = this;
+
+ $.fn.autogrow.resize(textarea);
+
+ $(textarea)
+ .focus(function() {
+ textarea.interval = setInterval(function() {
+ $.fn.autogrow.resize(textarea);
+ }, 500);
+ })
+ .blur(function() {
+ clearInterval(textarea.interval);
+ });
+ });
+ };
+
+ $.fn.autogrow.resize = function(textarea) {
+ var lineHeight = parseInt($(textarea).css('line-height'), 10);
+ var lines = textarea.value.split('\n');
+ var columns = textarea.cols;
+ var lineCount = 0;
+ $.each(lines, function() {
+ lineCount += Math.ceil(this.length / columns) || 1;
+ });
+ var height = lineHeight * (lineCount + 1);
+ $(textarea).css('height', height);
+ };
+})(jQuery);
+
+(function($) {
+ var comp, by;
+
+ function init() {
+ initEvents();
+ initComparator();
+ }
+
+ function initEvents() {
+ $(document).on("click", 'a.comment-close', function(event) {
+ event.preventDefault();
+ hide($(this).attr('id').substring(2));
+ });
+ $(document).on("click", 'a.vote', function(event) {
+ event.preventDefault();
+ handleVote($(this));
+ });
+ $(document).on("click", 'a.reply', function(event) {
+ event.preventDefault();
+ openReply($(this).attr('id').substring(2));
+ });
+ $(document).on("click", 'a.close-reply', function(event) {
+ event.preventDefault();
+ closeReply($(this).attr('id').substring(2));
+ });
+ $(document).on("click", 'a.sort-option', function(event) {
+ event.preventDefault();
+ handleReSort($(this));
+ });
+ $(document).on("click", 'a.show-proposal', function(event) {
+ event.preventDefault();
+ showProposal($(this).attr('id').substring(2));
+ });
+ $(document).on("click", 'a.hide-proposal', function(event) {
+ event.preventDefault();
+ hideProposal($(this).attr('id').substring(2));
+ });
+ $(document).on("click", 'a.show-propose-change', function(event) {
+ event.preventDefault();
+ showProposeChange($(this).attr('id').substring(2));
+ });
+ $(document).on("click", 'a.hide-propose-change', function(event) {
+ event.preventDefault();
+ hideProposeChange($(this).attr('id').substring(2));
+ });
+ $(document).on("click", 'a.accept-comment', function(event) {
+ event.preventDefault();
+ acceptComment($(this).attr('id').substring(2));
+ });
+ $(document).on("click", 'a.delete-comment', function(event) {
+ event.preventDefault();
+ deleteComment($(this).attr('id').substring(2));
+ });
+ $(document).on("click", 'a.comment-markup', function(event) {
+ event.preventDefault();
+ toggleCommentMarkupBox($(this).attr('id').substring(2));
+ });
+ }
+
+ /**
+ * Set comp, which is a comparator function used for sorting and
+ * inserting comments into the list.
+ */
+ function setComparator() {
+ // If the first three letters are "asc", sort in ascending order
+ // and remove the prefix.
+ if (by.substring(0,3) == 'asc') {
+ var i = by.substring(3);
+ comp = function(a, b) { return a[i] - b[i]; };
+ } else {
+ // Otherwise sort in descending order.
+ comp = function(a, b) { return b[by] - a[by]; };
+ }
+
+ // Reset link styles and format the selected sort option.
+ $('a.sel').attr('href', '#').removeClass('sel');
+ $('a.by' + by).removeAttr('href').addClass('sel');
+ }
+
+ /**
+ * Create a comp function. If the user has preferences stored in
+ * the sortBy cookie, use those, otherwise use the default.
+ */
+ function initComparator() {
+ by = 'rating'; // Default to sort by rating.
+ // If the sortBy cookie is set, use that instead.
+ if (document.cookie.length > 0) {
+ var start = document.cookie.indexOf('sortBy=');
+ if (start != -1) {
+ start = start + 7;
+ var end = document.cookie.indexOf(";", start);
+ if (end == -1) {
+ end = document.cookie.length;
+ by = unescape(document.cookie.substring(start, end));
+ }
+ }
+ }
+ setComparator();
+ }
+
+ /**
+ * Show a comment div.
+ */
+ function show(id) {
+ $('#ao' + id).hide();
+ $('#ah' + id).show();
+ var context = $.extend({id: id}, opts);
+ var popup = $(renderTemplate(popupTemplate, context)).hide();
+ popup.find('textarea[name="proposal"]').hide();
+ popup.find('a.by' + by).addClass('sel');
+ var form = popup.find('#cf' + id);
+ form.submit(function(event) {
+ event.preventDefault();
+ addComment(form);
+ });
+ $('#s' + id).after(popup);
+ popup.slideDown('fast', function() {
+ getComments(id);
+ });
+ }
+
+ /**
+ * Hide a comment div.
+ */
+ function hide(id) {
+ $('#ah' + id).hide();
+ $('#ao' + id).show();
+ var div = $('#sc' + id);
+ div.slideUp('fast', function() {
+ div.remove();
+ });
+ }
+
+ /**
+ * Perform an ajax request to get comments for a node
+ * and insert the comments into the comments tree.
+ */
+ function getComments(id) {
+ $.ajax({
+ type: 'GET',
+ url: opts.getCommentsURL,
+ data: {node: id},
+ success: function(data, textStatus, request) {
+ var ul = $('#cl' + id);
+ var speed = 100;
+ $('#cf' + id)
+ .find('textarea[name="proposal"]')
+ .data('source', data.source);
+
+ if (data.comments.length === 0) {
+ ul.html('<li>No comments yet.</li>');
+ ul.data('empty', true);
+ } else {
+ // If there are comments, sort them and put them in the list.
+ var comments = sortComments(data.comments);
+ speed = data.comments.length * 100;
+ appendComments(comments, ul);
+ ul.data('empty', false);
+ }
+ $('#cn' + id).slideUp(speed + 200);
+ ul.slideDown(speed);
+ },
+ error: function(request, textStatus, error) {
+ showError('Oops, there was a problem retrieving the comments.');
+ },
+ dataType: 'json'
+ });
+ }
+
+ /**
+ * Add a comment via ajax and insert the comment into the comment tree.
+ */
+ function addComment(form) {
+ var node_id = form.find('input[name="node"]').val();
+ var parent_id = form.find('input[name="parent"]').val();
+ var text = form.find('textarea[name="comment"]').val();
+ var proposal = form.find('textarea[name="proposal"]').val();
+
+ if (text == '') {
+ showError('Please enter a comment.');
+ return;
+ }
+
+ // Disable the form that is being submitted.
+ form.find('textarea,input').attr('disabled', 'disabled');
+
+ // Send the comment to the server.
+ $.ajax({
+ type: "POST",
+ url: opts.addCommentURL,
+ dataType: 'json',
+ data: {
+ node: node_id,
+ parent: parent_id,
+ text: text,
+ proposal: proposal
+ },
+ success: function(data, textStatus, error) {
+ // Reset the form.
+ if (node_id) {
+ hideProposeChange(node_id);
+ }
+ form.find('textarea')
+ .val('')
+ .add(form.find('input'))
+ .removeAttr('disabled');
+ var ul = $('#cl' + (node_id || parent_id));
+ if (ul.data('empty')) {
+ $(ul).empty();
+ ul.data('empty', false);
+ }
+ insertComment(data.comment);
+ var ao = $('#ao' + node_id);
+ ao.find('img').attr({'src': opts.commentBrightImage});
+ if (node_id) {
+ // if this was a "root" comment, remove the commenting box
+ // (the user can get it back by reopening the comment popup)
+ $('#ca' + node_id).slideUp();
+ }
+ },
+ error: function(request, textStatus, error) {
+ form.find('textarea,input').removeAttr('disabled');
+ showError('Oops, there was a problem adding the comment.');
+ }
+ });
+ }
+
+ /**
+ * Recursively append comments to the main comment list and children
+ * lists, creating the comment tree.
+ */
+ function appendComments(comments, ul) {
+ $.each(comments, function() {
+ var div = createCommentDiv(this);
+ ul.append($(document.createElement('li')).html(div));
+ appendComments(this.children, div.find('ul.comment-children'));
+ // To avoid stagnating data, don't store the comments children in data.
+ this.children = null;
+ div.data('comment', this);
+ });
+ }
+
+ /**
+ * After adding a new comment, it must be inserted in the correct
+ * location in the comment tree.
+ */
+ function insertComment(comment) {
+ var div = createCommentDiv(comment);
+
+ // To avoid stagnating data, don't store the comments children in data.
+ comment.children = null;
+ div.data('comment', comment);
+
+ var ul = $('#cl' + (comment.node || comment.parent));
+ var siblings = getChildren(ul);
+
+ var li = $(document.createElement('li'));
+ li.hide();
+
+ // Determine where in the parents children list to insert this comment.
+ for(i=0; i < siblings.length; i++) {
+ if (comp(comment, siblings[i]) <= 0) {
+ $('#cd' + siblings[i].id)
+ .parent()
+ .before(li.html(div));
+ li.slideDown('fast');
+ return;
+ }
+ }
+
+ // If we get here, this comment rates lower than all the others,
+ // or it is the only comment in the list.
+ ul.append(li.html(div));
+ li.slideDown('fast');
+ }
+
+ function acceptComment(id) {
+ $.ajax({
+ type: 'POST',
+ url: opts.acceptCommentURL,
+ data: {id: id},
+ success: function(data, textStatus, request) {
+ $('#cm' + id).fadeOut('fast');
+ $('#cd' + id).removeClass('moderate');
+ },
+ error: function(request, textStatus, error) {
+ showError('Oops, there was a problem accepting the comment.');
+ }
+ });
+ }
+
+ function deleteComment(id) {
+ $.ajax({
+ type: 'POST',
+ url: opts.deleteCommentURL,
+ data: {id: id},
+ success: function(data, textStatus, request) {
+ var div = $('#cd' + id);
+ if (data == 'delete') {
+ // Moderator mode: remove the comment and all children immediately
+ div.slideUp('fast', function() {
+ div.remove();
+ });
+ return;
+ }
+ // User mode: only mark the comment as deleted
+ div
+ .find('span.user-id:first')
+ .text('[deleted]').end()
+ .find('div.comment-text:first')
+ .text('[deleted]').end()
+ .find('#cm' + id + ', #dc' + id + ', #ac' + id + ', #rc' + id +
+ ', #sp' + id + ', #hp' + id + ', #cr' + id + ', #rl' + id)
+ .remove();
+ var comment = div.data('comment');
+ comment.username = '[deleted]';
+ comment.text = '[deleted]';
+ div.data('comment', comment);
+ },
+ error: function(request, textStatus, error) {
+ showError('Oops, there was a problem deleting the comment.');
+ }
+ });
+ }
+
+ function showProposal(id) {
+ $('#sp' + id).hide();
+ $('#hp' + id).show();
+ $('#pr' + id).slideDown('fast');
+ }
+
+ function hideProposal(id) {
+ $('#hp' + id).hide();
+ $('#sp' + id).show();
+ $('#pr' + id).slideUp('fast');
+ }
+
+ function showProposeChange(id) {
+ $('#pc' + id).hide();
+ $('#hc' + id).show();
+ var textarea = $('#pt' + id);
+ textarea.val(textarea.data('source'));
+ $.fn.autogrow.resize(textarea[0]);
+ textarea.slideDown('fast');
+ }
+
+ function hideProposeChange(id) {
+ $('#hc' + id).hide();
+ $('#pc' + id).show();
+ var textarea = $('#pt' + id);
+ textarea.val('').removeAttr('disabled');
+ textarea.slideUp('fast');
+ }
+
+ function toggleCommentMarkupBox(id) {
+ $('#mb' + id).toggle();
+ }
+
+ /** Handle when the user clicks on a sort by link. */
+ function handleReSort(link) {
+ var classes = link.attr('class').split(/\s+/);
+ for (var i=0; i<classes.length; i++) {
+ if (classes[i] != 'sort-option') {
+ by = classes[i].substring(2);
+ }
+ }
+ setComparator();
+ // Save/update the sortBy cookie.
+ var expiration = new Date();
+ expiration.setDate(expiration.getDate() + 365);
+ document.cookie= 'sortBy=' + escape(by) +
+ ';expires=' + expiration.toUTCString();
+ $('ul.comment-ul').each(function(index, ul) {
+ var comments = getChildren($(ul), true);
+ comments = sortComments(comments);
+ appendComments(comments, $(ul).empty());
+ });
+ }
+
+ /**
+ * Function to process a vote when a user clicks an arrow.
+ */
+ function handleVote(link) {
+ if (!opts.voting) {
+ showError("You'll need to login to vote.");
+ return;
+ }
+
+ var id = link.attr('id');
+ if (!id) {
+ // Didn't click on one of the voting arrows.
+ return;
+ }
+ // If it is an unvote, the new vote value is 0,
+ // Otherwise it's 1 for an upvote, or -1 for a downvote.
+ var value = 0;
+ if (id.charAt(1) != 'u') {
+ value = id.charAt(0) == 'u' ? 1 : -1;
+ }
+ // The data to be sent to the server.
+ var d = {
+ comment_id: id.substring(2),
+ value: value
+ };
+
+ // Swap the vote and unvote links.
+ link.hide();
+ $('#' + id.charAt(0) + (id.charAt(1) == 'u' ? 'v' : 'u') + d.comment_id)
+ .show();
+
+ // The div the comment is displayed in.
+ var div = $('div#cd' + d.comment_id);
+ var data = div.data('comment');
+
+ // If this is not an unvote, and the other vote arrow has
+ // already been pressed, unpress it.
+ if ((d.value !== 0) && (data.vote === d.value * -1)) {
+ $('#' + (d.value == 1 ? 'd' : 'u') + 'u' + d.comment_id).hide();
+ $('#' + (d.value == 1 ? 'd' : 'u') + 'v' + d.comment_id).show();
+ }
+
+ // Update the comments rating in the local data.
+ data.rating += (data.vote === 0) ? d.value : (d.value - data.vote);
+ data.vote = d.value;
+ div.data('comment', data);
+
+ // Change the rating text.
+ div.find('.rating:first')
+ .text(data.rating + ' point' + (data.rating == 1 ? '' : 's'));
+
+ // Send the vote information to the server.
+ $.ajax({
+ type: "POST",
+ url: opts.processVoteURL,
+ data: d,
+ error: function(request, textStatus, error) {
+ showError('Oops, there was a problem casting that vote.');
+ }
+ });
+ }
+
+ /**
+ * Open a reply form used to reply to an existing comment.
+ */
+ function openReply(id) {
+ // Swap out the reply link for the hide link
+ $('#rl' + id).hide();
+ $('#cr' + id).show();
+
+ // Add the reply li to the children ul.
+ var div = $(renderTemplate(replyTemplate, {id: id})).hide();
+ $('#cl' + id)
+ .prepend(div)
+ // Setup the submit handler for the reply form.
+ .find('#rf' + id)
+ .submit(function(event) {
+ event.preventDefault();
+ addComment($('#rf' + id));
+ closeReply(id);
+ })
+ .find('input[type=button]')
+ .click(function() {
+ closeReply(id);
+ });
+ div.slideDown('fast', function() {
+ $('#rf' + id).find('textarea').focus();
+ });
+ }
+
+ /**
+ * Close the reply form opened with openReply.
+ */
+ function closeReply(id) {
+ // Remove the reply div from the DOM.
+ $('#rd' + id).slideUp('fast', function() {
+ $(this).remove();
+ });
+
+ // Swap out the hide link for the reply link
+ $('#cr' + id).hide();
+ $('#rl' + id).show();
+ }
+
+ /**
+ * Recursively sort a tree of comments using the comp comparator.
+ */
+ function sortComments(comments) {
+ comments.sort(comp);
+ $.each(comments, function() {
+ this.children = sortComments(this.children);
+ });
+ return comments;
+ }
+
+ /**
+ * Get the children comments from a ul. If recursive is true,
+ * recursively include childrens' children.
+ */
+ function getChildren(ul, recursive) {
+ var children = [];
+ ul.children().children("[id^='cd']")
+ .each(function() {
+ var comment = $(this).data('comment');
+ if (recursive)
+ comment.children = getChildren($(this).find('#cl' + comment.id), true);
+ children.push(comment);
+ });
+ return children;
+ }
+
+ /** Create a div to display a comment in. */
+ function createCommentDiv(comment) {
+ if (!comment.displayed && !opts.moderator) {
+ return $('<div class="moderate">Thank you! Your comment will show up '
+ + 'once it is has been approved by a moderator.</div>');
+ }
+ // Prettify the comment rating.
+ comment.pretty_rating = comment.rating + ' point' +
+ (comment.rating == 1 ? '' : 's');
+ // Make a class (for displaying not yet moderated comments differently)
+ comment.css_class = comment.displayed ? '' : ' moderate';
+ // Create a div for this comment.
+ var context = $.extend({}, opts, comment);
+ var div = $(renderTemplate(commentTemplate, context));
+
+ // If the user has voted on this comment, highlight the correct arrow.
+ if (comment.vote) {
+ var direction = (comment.vote == 1) ? 'u' : 'd';
+ div.find('#' + direction + 'v' + comment.id).hide();
+ div.find('#' + direction + 'u' + comment.id).show();
+ }
+
+ if (opts.moderator || comment.text != '[deleted]') {
+ div.find('a.reply').show();
+ if (comment.proposal_diff)
+ div.find('#sp' + comment.id).show();
+ if (opts.moderator && !comment.displayed)
+ div.find('#cm' + comment.id).show();
+ if (opts.moderator || (opts.username == comment.username))
+ div.find('#dc' + comment.id).show();
+ }
+ return div;
+ }
+
+ /**
+ * A simple template renderer. Placeholders such as <%id%> are replaced
+ * by context['id'] with items being escaped. Placeholders such as <#id#>
+ * are not escaped.
+ */
+ function renderTemplate(template, context) {
+ var esc = $(document.createElement('div'));
+
+ function handle(ph, escape) {
+ var cur = context;
+ $.each(ph.split('.'), function() {
+ cur = cur[this];
+ });
+ return escape ? esc.text(cur || "").html() : cur;
+ }
+
+ return template.replace(/<([%#])([\w\.]*)\1>/g, function() {
+ return handle(arguments[2], arguments[1] == '%' ? true : false);
+ });
+ }
+
+ /** Flash an error message briefly. */
+ function showError(message) {
+ $(document.createElement('div')).attr({'class': 'popup-error'})
+ .append($(document.createElement('div'))
+ .attr({'class': 'error-message'}).text(message))
+ .appendTo('body')
+ .fadeIn("slow")
+ .delay(2000)
+ .fadeOut("slow");
+ }
+
+ /** Add a link the user uses to open the comments popup. */
+ $.fn.comment = function() {
+ return this.each(function() {
+ var id = $(this).attr('id').substring(1);
+ var count = COMMENT_METADATA[id];
+ var title = count + ' comment' + (count == 1 ? '' : 's');
+ var image = count > 0 ? opts.commentBrightImage : opts.commentImage;
+ var addcls = count == 0 ? ' nocomment' : '';
+ $(this)
+ .append(
+ $(document.createElement('a')).attr({
+ href: '#',
+ 'class': 'sphinx-comment-open' + addcls,
+ id: 'ao' + id
+ })
+ .append($(document.createElement('img')).attr({
+ src: image,
+ alt: 'comment',
+ title: title
+ }))
+ .click(function(event) {
+ event.preventDefault();
+ show($(this).attr('id').substring(2));
+ })
+ )
+ .append(
+ $(document.createElement('a')).attr({
+ href: '#',
+ 'class': 'sphinx-comment-close hidden',
+ id: 'ah' + id
+ })
+ .append($(document.createElement('img')).attr({
+ src: opts.closeCommentImage,
+ alt: 'close',
+ title: 'close'
+ }))
+ .click(function(event) {
+ event.preventDefault();
+ hide($(this).attr('id').substring(2));
+ })
+ );
+ });
+ };
+
+ var opts = {
+ processVoteURL: '/_process_vote',
+ addCommentURL: '/_add_comment',
+ getCommentsURL: '/_get_comments',
+ acceptCommentURL: '/_accept_comment',
+ deleteCommentURL: '/_delete_comment',
+ commentImage: '/static/_static/comment.png',
+ closeCommentImage: '/static/_static/comment-close.png',
+ loadingImage: '/static/_static/ajax-loader.gif',
+ commentBrightImage: '/static/_static/comment-bright.png',
+ upArrow: '/static/_static/up.png',
+ downArrow: '/static/_static/down.png',
+ upArrowPressed: '/static/_static/up-pressed.png',
+ downArrowPressed: '/static/_static/down-pressed.png',
+ voting: false,
+ moderator: false
+ };
+
+ if (typeof COMMENT_OPTIONS != "undefined") {
+ opts = jQuery.extend(opts, COMMENT_OPTIONS);
+ }
+
+ var popupTemplate = '\
+ <div class="sphinx-comments" id="sc<%id%>">\
+ <p class="sort-options">\
+ Sort by:\
+ <a href="#" class="sort-option byrating">best rated</a>\
+ <a href="#" class="sort-option byascage">newest</a>\
+ <a href="#" class="sort-option byage">oldest</a>\
+ </p>\
+ <div class="comment-header">Comments</div>\
+ <div class="comment-loading" id="cn<%id%>">\
+ loading comments... <img src="<%loadingImage%>" alt="" /></div>\
+ <ul id="cl<%id%>" class="comment-ul"></ul>\
+ <div id="ca<%id%>">\
+ <p class="add-a-comment">Add a comment\
+ (<a href="#" class="comment-markup" id="ab<%id%>">markup</a>):</p>\
+ <div class="comment-markup-box" id="mb<%id%>">\
+ reStructured text markup: <i>*emph*</i>, <b>**strong**</b>, \
+ <code>``code``</code>, \
+ code blocks: <code>::</code> and an indented block after blank line</div>\
+ <form method="post" id="cf<%id%>" class="comment-form" action="">\
+ <textarea name="comment" cols="80"></textarea>\
+ <p class="propose-button">\
+ <a href="#" id="pc<%id%>" class="show-propose-change">\
+ Propose a change ▹\
+ </a>\
+ <a href="#" id="hc<%id%>" class="hide-propose-change">\
+ Propose a change ▿\
+ </a>\
+ </p>\
+ <textarea name="proposal" id="pt<%id%>" cols="80"\
+ spellcheck="false"></textarea>\
+ <input type="submit" value="Add comment" />\
+ <input type="hidden" name="node" value="<%id%>" />\
+ <input type="hidden" name="parent" value="" />\
+ </form>\
+ </div>\
+ </div>';
+
+ var commentTemplate = '\
+ <div id="cd<%id%>" class="sphinx-comment<%css_class%>">\
+ <div class="vote">\
+ <div class="arrow">\
+ <a href="#" id="uv<%id%>" class="vote" title="vote up">\
+ <img src="<%upArrow%>" />\
+ </a>\
+ <a href="#" id="uu<%id%>" class="un vote" title="vote up">\
+ <img src="<%upArrowPressed%>" />\
+ </a>\
+ </div>\
+ <div class="arrow">\
+ <a href="#" id="dv<%id%>" class="vote" title="vote down">\
+ <img src="<%downArrow%>" id="da<%id%>" />\
+ </a>\
+ <a href="#" id="du<%id%>" class="un vote" title="vote down">\
+ <img src="<%downArrowPressed%>" />\
+ </a>\
+ </div>\
+ </div>\
+ <div class="comment-content">\
+ <p class="tagline comment">\
+ <span class="user-id"><%username%></span>\
+ <span class="rating"><%pretty_rating%></span>\
+ <span class="delta"><%time.delta%></span>\
+ </p>\
+ <div class="comment-text comment"><#text#></div>\
+ <p class="comment-opts comment">\
+ <a href="#" class="reply hidden" id="rl<%id%>">reply ▹</a>\
+ <a href="#" class="close-reply" id="cr<%id%>">reply ▿</a>\
+ <a href="#" id="sp<%id%>" class="show-proposal">proposal ▹</a>\
+ <a href="#" id="hp<%id%>" class="hide-proposal">proposal ▿</a>\
+ <a href="#" id="dc<%id%>" class="delete-comment hidden">delete</a>\
+ <span id="cm<%id%>" class="moderation hidden">\
+ <a href="#" id="ac<%id%>" class="accept-comment">accept</a>\
+ </span>\
+ </p>\
+ <pre class="proposal" id="pr<%id%>">\
+<#proposal_diff#>\
+ </pre>\
+ <ul class="comment-children" id="cl<%id%>"></ul>\
+ </div>\
+ <div class="clearleft"></div>\
+ </div>\
+ </div>';
+
+ var replyTemplate = '\
+ <li>\
+ <div class="reply-div" id="rd<%id%>">\
+ <form id="rf<%id%>">\
+ <textarea name="comment" cols="80"></textarea>\
+ <input type="submit" value="Add reply" />\
+ <input type="button" value="Cancel" />\
+ <input type="hidden" name="parent" value="<%id%>" />\
+ <input type="hidden" name="node" value="" />\
+ </form>\
+ </div>\
+ </li>';
+
+ $(document).ready(function() {
+ init();
+ });
+})(jQuery);
+
+$(document).ready(function() {
+ // add comment anchors for all paragraphs that are commentable
+ $('.sphinx-has-comment').comment();
+
+ // highlight search words in search results
+ $("div.context").each(function() {
+ var params = $.getQueryParameters();
+ var terms = (params.q) ? params.q[0].split(/\s+/) : [];
+ var result = $(this);
+ $.each(terms, function() {
+ result.highlightText(this.toLowerCase(), 'highlighted');
+ });
+ });
+
+ // directly open comment window if requested
+ var anchor = document.location.hash;
+ if (anchor.substring(0, 9) == '#comment-') {
+ $('#ao' + anchor.substring(9)).click();
+ document.location.hash = '#s' + anchor.substring(9);
+ }
+});
[04/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/faq.html
----------------------------------------------------------------------
diff --git a/faq.html b/faq.html
new file mode 100644
index 0000000..d5e001d
--- /dev/null
+++ b/faq.html
@@ -0,0 +1,293 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>FAQ — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="API Reference" href="code.html"/>
+ <link rel="prev" title="Security" href="security.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>FAQ</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/faq.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="faq">
+<h1>FAQ<a class="headerlink" href="#faq" title="Permalink to this headline">�</a></h1>
+<p><strong>Why isn’t my task getting scheduled?</strong></p>
+<p>There are very many reasons why your task might not be getting scheduled.
+Here are some of the common causes:</p>
+<ul class="simple">
+<li>Does your script “compile”, can the Airflow engine parse it and find your
+DAG object. To test this, you can run <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">list_dags</span></code> and
+confirm that your DAG shows up in the list. You can also run
+<code class="docutils literal"><span class="pre">airflow</span> <span class="pre">list_tasks</span> <span class="pre">foo_dag_id</span> <span class="pre">--tree</span></code> and confirm that your task
+shows up in the list as expected. If you use the CeleryExecutor, you
+may way to confirm that this works both where the scheduler runs as well
+as where the worker runs.</li>
+<li>Is your <code class="docutils literal"><span class="pre">start_date</span></code> set properly? The Airflow scheduler triggers the
+task soon after the <code class="docutils literal"><span class="pre">start_date</span> <span class="pre">+</span> <span class="pre">scheduler_interval</span></code> is passed.</li>
+<li>Is your <code class="docutils literal"><span class="pre">start_date</span></code> beyond where you can see it in the UI? If you
+set your it to some time say 3 months ago, you won’t be able to see
+it in the main view in the UI, but you should be able to see it in the
+<code class="docutils literal"><span class="pre">Menu</span> <span class="pre">-></span> <span class="pre">Browse</span> <span class="pre">->Task</span> <span class="pre">Instances</span></code>.</li>
+<li>Are the dependencies for the task met. The task instances directly
+upstream from the task need to be in a <code class="docutils literal"><span class="pre">success</span></code> state. Also,
+if you have set <code class="docutils literal"><span class="pre">depends_on_past=True</span></code>, the previous task instance
+needs to have succeeded (except if it is the first run for that task).
+Also, if <code class="docutils literal"><span class="pre">wait_for_downstream=True</span></code>, make sure you understand
+what it means.
+You can view how these properties are set from the <code class="docutils literal"><span class="pre">Task</span> <span class="pre">Details</span></code>
+page for your task.</li>
+<li>Are the DagRuns you need created and active? A DagRun represents a specific
+execution of an entire DAG and has a state (running, success, failed, ...).
+The scheduler creates new DagRun as it moves forward, but never goes back
+in time to create new ones. The scheduler only evaluates <code class="docutils literal"><span class="pre">running</span></code> DagRuns
+to see what task instances it can trigger. Note that clearing tasks
+instances (from the UI or CLI) does set the state of a DagRun back to
+running. You can bulk view the list of DagRuns and alter states by clicking
+on the schedule tag for a DAG.</li>
+<li>Is the <code class="docutils literal"><span class="pre">concurrency</span></code> parameter of your DAG reached? <code class="docutils literal"><span class="pre">concurency</span></code> defines
+how many <code class="docutils literal"><span class="pre">running</span></code> task instances a DAG is allowed to have, beyond which
+point things get queued.</li>
+<li>Is the <code class="docutils literal"><span class="pre">max_active_runs</span></code> parameter of your DAG reached? <code class="docutils literal"><span class="pre">max_active_runs</span></code> defines
+how many <code class="docutils literal"><span class="pre">running</span></code> concurrent instances of a DAG there are allowed to be.</li>
+</ul>
+<p>You may also want to read the Scheduler section of the docs and make
+sure you fully understand how it proceeds.</p>
+<p><strong>How do I trigger tasks based on another task’s failure?</strong></p>
+<p>Check out the <code class="docutils literal"><span class="pre">Trigger</span> <span class="pre">Rule</span></code> section in the Concepts section of the
+documentation</p>
+<p><strong>Why are connection passwords still not encrypted in the metadata db after I installed airflow[crypto]</strong>?</p>
+<ul class="simple">
+<li>Verify that the <code class="docutils literal"><span class="pre">fernet_key</span></code> defined in <code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code> is a valid Fernet key. It must be a base64-encoded 32-byte key. You need to restart the webserver after you update the key</li>
+<li>For existing connections (the ones that you had defined before installing <code class="docutils literal"><span class="pre">airflow[crypto]</span></code> and creating a Fernet key), you need to open each connection in the connection admin UI, re-type the password, and save it</li>
+</ul>
+<p><strong>What’s the deal with ``start_date``?</strong></p>
+<p><code class="docutils literal"><span class="pre">start_date</span></code> is partly legacy from the pre-DagRun era, but it is still
+relevant in many ways. When creating a new DAG, you probably want to set
+a global <code class="docutils literal"><span class="pre">start_date</span></code> for your tasks using <code class="docutils literal"><span class="pre">default_args</span></code>. The first
+DagRun to be created will be based on the <code class="docutils literal"><span class="pre">min(start_date)</span></code> for all your
+task. From that point on, the scheduler creates new DagRuns based on
+your <code class="docutils literal"><span class="pre">schedule_interval</span></code> and the corresponding task instances run as your
+dependencies are met. When introducing new tasks to your DAG, you need to
+pay special attention to <code class="docutils literal"><span class="pre">start_date</span></code>, and may want to reactivate
+inactive DagRuns to get the new task to get onboarded properly.</p>
+<p>We recommend against using dynamic values as <code class="docutils literal"><span class="pre">start_date</span></code>, especially
+<code class="docutils literal"><span class="pre">datetime.now()</span></code> as it can be quite confusing. The task is triggered
+once the period closes, and in theory an <code class="docutils literal"><span class="pre">@hourly</span></code> DAG would never get to
+an hour after now as <code class="docutils literal"><span class="pre">now()</span></code> moves along.</p>
+<p>We also recommend using rounded <code class="docutils literal"><span class="pre">start_date</span></code> in relation to your
+<code class="docutils literal"><span class="pre">schedule_interval</span></code>. This means an <code class="docutils literal"><span class="pre">@hourly</span></code> would be at <code class="docutils literal"><span class="pre">00:00</span></code>
+minutes:seconds, a <code class="docutils literal"><span class="pre">@daily</span></code> job at midnight, a <code class="docutils literal"><span class="pre">@monthly</span></code> job on the
+first of the month. You can use any sensor or a <code class="docutils literal"><span class="pre">TimeDeltaSensor</span></code> to delay
+the execution of tasks within that period. While <code class="docutils literal"><span class="pre">schedule_interval</span></code>
+does allow specifying a <code class="docutils literal"><span class="pre">datetime.timedelta</span></code>
+object, we recommend using the macros or cron expressions instead, as
+it enforces this idea of rounded schedules.</p>
+<p>When using <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> it’s important to pay special attention
+to <code class="docutils literal"><span class="pre">start_date</span></code> as the past dependency is not enforced only on the specific
+schedule of the <code class="docutils literal"><span class="pre">start_date</span></code> specified for the task. It’ also
+important to watch DagRun activity status in time when introducing
+new <code class="docutils literal"><span class="pre">depends_on_past=True</span></code>, unless you are planning on running a backfill
+for the new task(s).</p>
+<p>Also important to note is that the tasks <code class="docutils literal"><span class="pre">start_date</span></code>, in the context of a
+backfill CLI command, get overridden by the backfill’s command <code class="docutils literal"><span class="pre">start_date</span></code>.
+This allows for a backfill on tasks that have <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> to
+actually start, if it wasn’t the case, the backfill just wouldn’t start.</p>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="code.html" class="btn btn-neutral float-right" title="API Reference" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="security.html" class="btn btn-neutral" title="Security" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/genindex.html
----------------------------------------------------------------------
diff --git a/genindex.html b/genindex.html
new file mode 100644
index 0000000..eea3c14
--- /dev/null
+++ b/genindex.html
@@ -0,0 +1,1258 @@
+
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Index — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li></li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+
+<h1 id="index">Index</h1>
+
+<div class="genindex-jumpbox">
+ <a href="#A"><strong>A</strong></a>
+ | <a href="#B"><strong>B</strong></a>
+ | <a href="#C"><strong>C</strong></a>
+ | <a href="#D"><strong>D</strong></a>
+ | <a href="#E"><strong>E</strong></a>
+ | <a href="#F"><strong>F</strong></a>
+ | <a href="#G"><strong>G</strong></a>
+ | <a href="#H"><strong>H</strong></a>
+ | <a href="#I"><strong>I</strong></a>
+ | <a href="#K"><strong>K</strong></a>
+ | <a href="#L"><strong>L</strong></a>
+ | <a href="#M"><strong>M</strong></a>
+ | <a href="#O"><strong>O</strong></a>
+ | <a href="#P"><strong>P</strong></a>
+ | <a href="#R"><strong>R</strong></a>
+ | <a href="#S"><strong>S</strong></a>
+ | <a href="#T"><strong>T</strong></a>
+ | <a href="#U"><strong>U</strong></a>
+ | <a href="#V"><strong>V</strong></a>
+ | <a href="#W"><strong>W</strong></a>
+ | <a href="#X"><strong>X</strong></a>
+
+</div>
+<h2 id="A">A</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.DAG.add_task">add_task() (airflow.models.DAG method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DAG.add_tasks">add_tasks() (airflow.models.DAG method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#module-airflow.contrib.executors">airflow.contrib.executors (module)</a>
+ </dt>
+
+
+ <dt><a href="code.html#module-airflow.contrib.hooks">airflow.contrib.hooks (module)</a>
+ </dt>
+
+
+ <dt><a href="code.html#module-airflow.contrib.operators">airflow.contrib.operators (module)</a>
+ </dt>
+
+
+ <dt><a href="code.html#module-airflow.executors">airflow.executors (module)</a>
+ </dt>
+
+
+ <dt><a href="code.html#module-airflow.hooks">airflow.hooks (module)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#module-airflow.macros">airflow.macros (module)</a>
+ </dt>
+
+
+ <dt><a href="code.html#module-airflow.macros.hive">airflow.macros.hive (module)</a>
+ </dt>
+
+
+ <dt><a href="code.html#module-airflow.models">airflow.models (module)</a>
+ </dt>
+
+
+ <dt><a href="code.html#module-airflow.operators">airflow.operators (module)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.TaskInstance.are_dependencies_met">are_dependencies_met() (airflow.models.TaskInstance method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.TaskInstance.are_dependents_done">are_dependents_done() (airflow.models.TaskInstance method)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="B">B</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.DagBag.bag_dag">bag_dag() (airflow.models.DagBag method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.BaseOperator">BaseOperator (class in airflow.models)</a>, <a href="code.html#airflow.models.BaseOperator">[1]</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.sensors.BaseSensorOperator">BaseSensorOperator (class in airflow.operators.sensors)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.BashOperator">BashOperator (class in airflow.operators)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.operators.BranchPythonOperator">BranchPythonOperator (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.DbApiHook.bulk_dump">bulk_dump() (airflow.hooks.DbApiHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.DbApiHook.bulk_load">bulk_load() (airflow.hooks.DbApiHook method)</a>
+ </dt>
+
+ <dd><dl>
+
+ <dt><a href="code.html#airflow.hooks.MySqlHook.bulk_load">(airflow.hooks.MySqlHook method)</a>
+ </dt>
+
+ </dl></dd>
+ </dl></td>
+</tr></table>
+
+<h2 id="C">C</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.executors.CeleryExecutor">CeleryExecutor (class in airflow.executors)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.S3Hook.check_for_bucket">check_for_bucket() (airflow.hooks.S3Hook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.S3Hook.check_for_key">check_for_key() (airflow.hooks.S3Hook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.HiveMetastoreHook.check_for_partition">check_for_partition() (airflow.hooks.HiveMetastoreHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.WebHDFSHook.check_for_path">check_for_path() (airflow.hooks.WebHDFSHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.S3Hook.check_for_prefix">check_for_prefix() (airflow.hooks.S3Hook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.S3Hook.check_for_wildcard_key">check_for_wildcard_key() (airflow.hooks.S3Hook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.hooks.SSHHook.check_output">check_output() (airflow.contrib.hooks.SSHHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.BaseOperator.clear">clear() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.TaskInstance.clear_xcom_data">clear_xcom_data() (airflow.models.TaskInstance method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DAG.cli">cli() (airflow.models.DAG method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.hooks.FTPHook.close_conn">close_conn() (airflow.contrib.hooks.FTPHook method)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.macros.hive.closest_ds_partition">closest_ds_partition() (in module airflow.macros.hive)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.hooks.CloudantHook">CloudantHook (class in airflow.contrib.hooks)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DagBag.collect_dags">collect_dags() (airflow.models.DagBag method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.TaskInstance.command">command() (airflow.models.TaskInstance method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DAG.concurrency_reached">concurrency_reached (airflow.models.DAG attribute)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.Connection">Connection (class in airflow.models)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.SlackAPIOperator.construct_api_call_params">construct_api_call_params() (airflow.operators.SlackAPIOperator method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.DruidHook.construct_ingest_query">construct_ingest_query() (airflow.hooks.DruidHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DAG.crawl_for_tasks">crawl_for_tasks() (airflow.models.DAG method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DAG.create_dagrun">create_dagrun() (airflow.models.DAG method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.hooks.FTPHook.create_directory">create_directory() (airflow.contrib.hooks.FTPHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.TaskInstance.current_state">current_state() (airflow.models.TaskInstance method)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="D">D</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.BaseOperator.dag">dag (airflow.models.BaseOperator attribute)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DAG">DAG (class in airflow.models)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DagBag">DagBag (class in airflow.models)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DagBag.dagbag_report">dagbag_report() (airflow.models.DagBag method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.hooks.CloudantHook.db">db() (airflow.contrib.hooks.CloudantHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.DbApiHook">DbApiHook (class in airflow.hooks)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.hooks.FTPHook.delete_directory">delete_directory() (airflow.contrib.hooks.FTPHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.hooks.FTPHook.delete_file">delete_file() (airflow.contrib.hooks.FTPHook method)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.contrib.hooks.FTPHook.describe_directory">describe_directory() (airflow.contrib.hooks.FTPHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.BaseOperator.detect_downstream_cycle">detect_downstream_cycle() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.docker_operator.DockerOperator">DockerOperator (class in airflow.operators.docker_operator)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.BaseOperator.downstream_list">downstream_list (airflow.models.BaseOperator attribute)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.DruidHook">DruidHook (class in airflow.hooks)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.macros.ds_add">ds_add() (in module airflow.macros)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.macros.ds_format">ds_format() (in module airflow.macros)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.DummyOperator">DummyOperator (class in airflow.operators)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="E">E</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.operators.EmailOperator">EmailOperator (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.TaskInstance.error">error() (airflow.models.TaskInstance method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.TaskInstance.evaluate_trigger_rule">evaluate_trigger_rule() (airflow.models.TaskInstance method)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.BaseOperator.execute">execute() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+ <dd><dl>
+
+ <dt><a href="code.html#airflow.operators.BashOperator.execute">(airflow.operators.BashOperator method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.SlackAPIOperator.execute">(airflow.operators.SlackAPIOperator method)</a>
+ </dt>
+
+ </dl></dd>
+
+ <dt><a href="code.html#airflow.operators.ExternalTaskSensor">ExternalTaskSensor (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.Connection.extra_dejson">extra_dejson (airflow.models.Connection attribute)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="F">F</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.DAG.filepath">filepath (airflow.models.DAG attribute)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DAG.folder">folder (airflow.models.DAG attribute)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.contrib.hooks.FTPHook">FTPHook (class in airflow.contrib.hooks)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="G">G</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.operators.GenericTransfer">GenericTransfer (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.S3Hook.get_bucket">get_bucket() (airflow.hooks.S3Hook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.hooks.FTPHook.get_conn">get_conn() (airflow.contrib.hooks.FTPHook method)</a>
+ </dt>
+
+ <dd><dl>
+
+ <dt><a href="code.html#airflow.contrib.hooks.VerticaHook.get_conn">(airflow.contrib.hooks.VerticaHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.DbApiHook.get_conn">(airflow.hooks.DbApiHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.DruidHook.get_conn">(airflow.hooks.DruidHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.HttpHook.get_conn">(airflow.hooks.HttpHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.MsSqlHook.get_conn">(airflow.hooks.MsSqlHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.MySqlHook.get_conn">(airflow.hooks.MySqlHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.PrestoHook.get_conn">(airflow.hooks.PrestoHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.S3Hook.get_conn">(airflow.hooks.S3Hook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.SqliteHook.get_conn">(airflow.hooks.SqliteHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.WebHDFSHook.get_conn">(airflow.hooks.WebHDFSHook method)</a>
+ </dt>
+
+ </dl></dd>
+
+ <dt><a href="code.html#airflow.hooks.DbApiHook.get_cursor">get_cursor() (airflow.hooks.DbApiHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DagBag.get_dag">get_dag() (airflow.models.DagBag method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.HiveMetastoreHook.get_databases">get_databases() (airflow.hooks.HiveMetastoreHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.BaseOperator.get_direct_relatives">get_direct_relatives() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.DbApiHook.get_first">get_first() (airflow.hooks.DbApiHook method)</a>
+ </dt>
+
+ <dd><dl>
+
+ <dt><a href="code.html#airflow.hooks.PrestoHook.get_first">(airflow.hooks.PrestoHook method)</a>
+ </dt>
+
+ </dl></dd>
+
+ <dt><a href="code.html#airflow.models.BaseOperator.get_flat_relatives">get_flat_relatives() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.S3Hook.get_key">get_key() (airflow.hooks.S3Hook method)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.hooks.HiveMetastoreHook.get_metastore_client">get_metastore_client() (airflow.hooks.HiveMetastoreHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.DbApiHook.get_pandas_df">get_pandas_df() (airflow.hooks.DbApiHook method)</a>
+ </dt>
+
+ <dd><dl>
+
+ <dt><a href="code.html#airflow.hooks.HiveServer2Hook.get_pandas_df">(airflow.hooks.HiveServer2Hook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.PrestoHook.get_pandas_df">(airflow.hooks.PrestoHook method)</a>
+ </dt>
+
+ </dl></dd>
+
+ <dt><a href="code.html#airflow.hooks.HiveMetastoreHook.get_partitions">get_partitions() (airflow.hooks.HiveMetastoreHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.DbApiHook.get_records">get_records() (airflow.hooks.DbApiHook method)</a>
+ </dt>
+
+ <dd><dl>
+
+ <dt><a href="code.html#airflow.hooks.HiveServer2Hook.get_records">(airflow.hooks.HiveServer2Hook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.PrestoHook.get_records">(airflow.hooks.PrestoHook method)</a>
+ </dt>
+
+ </dl></dd>
+
+ <dt><a href="code.html#airflow.hooks.HiveMetastoreHook.get_table">get_table() (airflow.hooks.HiveMetastoreHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.HiveMetastoreHook.get_tables">get_tables() (airflow.hooks.HiveMetastoreHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.BaseOperator.get_task_instances">get_task_instances() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DAG.get_template_env">get_template_env() (airflow.models.DAG method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.S3Hook.get_wildcard_key">get_wildcard_key() (airflow.hooks.S3Hook method)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="H">H</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.BaseOperator.has_dag">has_dag() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.HdfsSensor">HdfsSensor (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.operators.hipchat_operator.HipChatAPIOperator">HipChatAPIOperator (class in airflow.contrib.operators.hipchat_operator)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.operators.hipchat_operator.HipChatAPISendRoomNotificationOperator">HipChatAPISendRoomNotificationOperator (class in airflow.contrib.operators.hipchat_operator)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.Hive2SambaOperator">Hive2SambaOperator (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.HiveCliHook">HiveCliHook (class in airflow.hooks)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.HiveMetastoreHook">HiveMetastoreHook (class in airflow.hooks)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.operators.HiveOperator">HiveOperator (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.HivePartitionSensor">HivePartitionSensor (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.HiveServer2Hook">HiveServer2Hook (class in airflow.hooks)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.HiveToDruidTransfer">HiveToDruidTransfer (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.HiveToMySqlTransfer">HiveToMySqlTransfer (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.HttpHook">HttpHook (class in airflow.hooks)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.HttpSensor">HttpSensor (class in airflow.operators)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="I">I</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.hooks.DbApiHook.insert_rows">insert_rows() (airflow.hooks.DbApiHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.macros.integrate_plugins">integrate_plugins() (in module airflow.macros)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DAG.is_paused">is_paused (airflow.models.DAG attribute)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.TaskInstance.is_premature">is_premature() (airflow.models.TaskInstance method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.TaskInstance.is_queueable">is_queueable() (airflow.models.TaskInstance method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.TaskInstance.is_runnable">is_runnable() (airflow.models.TaskInstance method)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="K">K</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.TaskInstance.key">key (airflow.models.TaskInstance attribute)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.DagBag.kill_zombies">kill_zombies() (airflow.models.DagBag method)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="L">L</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.DAG.latest_execution_date">latest_execution_date (airflow.models.DAG attribute)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.hooks.FTPHook.list_directory">list_directory() (airflow.contrib.hooks.FTPHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.S3Hook.list_keys">list_keys() (airflow.hooks.S3Hook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.S3Hook.list_prefixes">list_prefixes() (airflow.hooks.S3Hook method)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.hooks.HiveCliHook.load_file">load_file() (airflow.hooks.HiveCliHook method)</a>
+ </dt>
+
+ <dd><dl>
+
+ <dt><a href="code.html#airflow.hooks.S3Hook.load_file">(airflow.hooks.S3Hook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.WebHDFSHook.load_file">(airflow.hooks.WebHDFSHook method)</a>
+ </dt>
+
+ </dl></dd>
+
+ <dt><a href="code.html#airflow.hooks.DruidHook.load_from_hdfs">load_from_hdfs() (airflow.hooks.DruidHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.S3Hook.load_string">load_string() (airflow.hooks.S3Hook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.executors.LocalExecutor">LocalExecutor (class in airflow.executors)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="M">M</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.hooks.HiveMetastoreHook.max_partition">max_partition() (airflow.hooks.HiveMetastoreHook method)</a>
+ </dt>
+
+ <dd><dl>
+
+ <dt><a href="code.html#airflow.macros.hive.max_partition">(in module airflow.macros.hive)</a>
+ </dt>
+
+ </dl></dd>
+
+ <dt><a href="code.html#airflow.operators.MetastorePartitionSensor">MetastorePartitionSensor (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.MsSqlHook">MsSqlHook (class in airflow.hooks)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.MsSqlOperator">MsSqlOperator (class in airflow.operators)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.operators.MsSqlToHiveTransfer">MsSqlToHiveTransfer (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.MySqlHook">MySqlHook (class in airflow.hooks)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.MySqlOperator">MySqlOperator (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.MySqlToHiveTransfer">MySqlToHiveTransfer (class in airflow.operators)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="O">O</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.BaseOperator.on_kill">on_kill() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="P">P</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.TaskInstance.pool_full">pool_full() (airflow.models.TaskInstance method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.hooks.SSHHook.Popen">Popen() (airflow.contrib.hooks.SSHHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.BaseOperator.post_execute">post_execute() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.PostgresHook">PostgresHook (class in airflow.hooks)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.PostgresOperator">PostgresOperator (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.BaseOperator.pre_execute">pre_execute() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.BaseOperator.prepare_template">prepare_template() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.operators.PrestoCheckOperator">PrestoCheckOperator (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.PrestoHook">PrestoHook (class in airflow.hooks)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.PrestoIntervalCheckOperator">PrestoIntervalCheckOperator (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.PrestoValueCheckOperator">PrestoValueCheckOperator (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DagBag.process_file">process_file() (airflow.models.DagBag method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.PythonOperator">PythonOperator (class in airflow.operators)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="R">R</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.macros.random">random() (in module airflow.macros)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.TaskInstance.ready_for_retry">ready_for_retry() (airflow.models.TaskInstance method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.TaskInstance.refresh_from_db">refresh_from_db() (airflow.models.TaskInstance method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.BaseOperator.render_template">render_template() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.BaseOperator.render_template_from_field">render_template_from_field() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.contrib.hooks.FTPHook.retrieve_file">retrieve_file() (airflow.contrib.hooks.FTPHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.DbApiHook.run">run() (airflow.hooks.DbApiHook method)</a>
+ </dt>
+
+ <dd><dl>
+
+ <dt><a href="code.html#airflow.hooks.HttpHook.run">(airflow.hooks.HttpHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.PrestoHook.run">(airflow.hooks.PrestoHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.BaseOperator.run">(airflow.models.BaseOperator method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DAG.run">(airflow.models.DAG method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.TaskInstance.run">(airflow.models.TaskInstance method)</a>
+ </dt>
+
+ </dl></dd>
+
+ <dt><a href="code.html#airflow.hooks.HttpHook.run_and_check">run_and_check() (airflow.hooks.HttpHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.HiveCliHook.run_cli">run_cli() (airflow.hooks.HiveCliHook method)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="S">S</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.hooks.S3Hook">S3Hook (class in airflow.hooks)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.S3KeySensor">S3KeySensor (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.S3ToHiveTransfer">S3ToHiveTransfer (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.BaseOperator.schedule_interval">schedule_interval (airflow.models.BaseOperator attribute)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.executors.SequentialExecutor">SequentialExecutor (class in airflow.executors)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DAG.set_dependency">set_dependency() (airflow.models.DAG method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.BaseOperator.set_downstream">set_downstream() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.BaseOperator.set_upstream">set_upstream() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.ShortCircuitOperator">ShortCircuitOperator (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.SimpleHttpOperator">SimpleHttpOperator (class in airflow.operators)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.DagBag.size">size() (airflow.models.DagBag method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.SlackAPIOperator">SlackAPIOperator (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.SlackAPIPostOperator">SlackAPIPostOperator (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.SqliteHook">SqliteHook (class in airflow.hooks)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.SqlSensor">SqlSensor (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.operators.SSHExecuteOperator">SSHExecuteOperator (class in airflow.contrib.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.hooks.SSHHook">SSHHook (class in airflow.contrib.hooks)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.hooks.FTPHook.store_file">store_file() (airflow.contrib.hooks.FTPHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DAG.sub_dag">sub_dag() (airflow.models.DAG method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.DAG.subdags">subdags (airflow.models.DAG attribute)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="T">T</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.hooks.HiveMetastoreHook.table_exists">table_exists() (airflow.hooks.HiveMetastoreHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.models.TaskInstance">TaskInstance (class in airflow.models)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.hooks.HiveCliHook.test_hql">test_hql() (airflow.hooks.HiveCliHook method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.TimeSensor">TimeSensor (class in airflow.operators)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.DAG.tree_view">tree_view() (airflow.models.DAG method)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.operators.TriggerDagRunOperator">TriggerDagRunOperator (class in airflow.operators)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.hooks.SSHHook.tunnel">tunnel() (airflow.contrib.hooks.SSHHook method)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="U">U</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.BaseOperator.upstream_list">upstream_list (airflow.models.BaseOperator attribute)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="V">V</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.contrib.hooks.VerticaHook">VerticaHook (class in airflow.contrib.hooks)</a>
+ </dt>
+
+
+ <dt><a href="code.html#airflow.contrib.operators.VerticaOperator">VerticaOperator (class in airflow.contrib.operators)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.contrib.operators.VerticaToHiveTransfer">VerticaToHiveTransfer (class in airflow.contrib.operators)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="W">W</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.hooks.WebHDFSHook">WebHDFSHook (class in airflow.hooks)</a>
+ </dt>
+
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.operators.WebHdfsSensor">WebHdfsSensor (class in airflow.operators)</a>
+ </dt>
+
+ </dl></td>
+</tr></table>
+
+<h2 id="X">X</h2>
+<table style="width: 100%" class="indextable genindextable"><tr>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.BaseOperator.xcom_pull">xcom_pull() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+ <dd><dl>
+
+ <dt><a href="code.html#airflow.models.TaskInstance.xcom_pull">(airflow.models.TaskInstance method)</a>
+ </dt>
+
+ </dl></dd>
+ </dl></td>
+ <td style="width: 33%" valign="top"><dl>
+
+ <dt><a href="code.html#airflow.models.BaseOperator.xcom_push">xcom_push() (airflow.models.BaseOperator method)</a>
+ </dt>
+
+ <dd><dl>
+
+ <dt><a href="code.html#airflow.models.TaskInstance.xcom_push">(airflow.models.TaskInstance method)</a>
+ </dt>
+
+ </dl></dd>
+ </dl></td>
+</tr></table>
+
+
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/index.html
----------------------------------------------------------------------
diff --git a/index.html b/index.html
new file mode 100644
index 0000000..c764349
--- /dev/null
+++ b/index.html
@@ -0,0 +1,417 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Apache Airflow (incubating) Documentation — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="#"/>
+ <link rel="next" title="Project" href="project.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="#" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="#">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="#">Docs</a> »</li>
+
+ <li>Apache Airflow (incubating) Documentation</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/index.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <a class="reference internal image-reference" href="_images/pin_large.png"><img alt="_images/pin_large.png" src="_images/pin_large.png" style="width: 100px;" /></a>
+<a class="reference internal image-reference" href="_images/incubator.jpg"><img alt="_images/incubator.jpg" src="_images/incubator.jpg" style="width: 150px;" /></a>
+<div class="section" id="apache-airflow-incubating-documentation">
+<h1>Apache Airflow (incubating) Documentation<a class="headerlink" href="#apache-airflow-incubating-documentation" title="Permalink to this headline">�</a></h1>
+<p>Airflow is a platform to programmatically author, schedule and monitor
+workflows.</p>
+<p>Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks.
+The airflow scheduler executes your tasks on an array of workers while
+following the specified dependencies. Rich command line utilities make
+performing complex surgeries on DAGs a snap. The rich user interface
+makes it easy to visualize pipelines running in production,
+monitor progress, and troubleshoot issues when needed.</p>
+<p>When workflows are defined as code, they become more maintainable,
+versionable, testable, and collaborative.</p>
+<hr class="docutils" />
+<img alt="_images/airflow.gif" src="_images/airflow.gif" />
+<hr class="docutils" />
+<div class="section" id="principles">
+<h2>Principles<a class="headerlink" href="#principles" title="Permalink to this headline">�</a></h2>
+<ul class="simple">
+<li><strong>Dynamic</strong>: Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically.</li>
+<li><strong>Extensible</strong>: Easily define your own operators, executors and extend the library so that it fits the level of abstraction that suits your environment.</li>
+<li><strong>Elegant</strong>: Airflow pipelines are lean and explicit. Parameterizing your scripts is built into the core of Airflow using the powerful <strong>Jinja</strong> templating engine.</li>
+<li><strong>Scalable</strong>: Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to infinity.</li>
+</ul>
+</div>
+<div class="section" id="beyond-the-horizon">
+<h2>Beyond the Horizon<a class="headerlink" href="#beyond-the-horizon" title="Permalink to this headline">�</a></h2>
+<p>Airflow <strong>is not</strong> a data streaming solution. Tasks do not move data from
+one to the other (though tasks can exchange metadata!). Airflow is not
+in the <a class="reference external" href="http://spark.apache.org/streaming/">Spark Streaming</a>
+or <a class="reference external" href="https://storm.apache.org/">Storm</a> space, it is more comparable to
+<a class="reference external" href="http://oozie.apache.org/">Oozie</a> or
+<a class="reference external" href="http://data.linkedin.com/opensource/azkaban">Azkaban</a>.</p>
+<p>Workflows are expected to be mostly static or slowly changing. You can think
+of the structure of the tasks in your workflow as slightly more dynamic
+than a database structure would be. Airflow workflows are expected to look
+similar from a run to the next, this allows for clarity around
+unit of work and continuity.</p>
+</div>
+<div class="section" id="content">
+<h2>Content<a class="headerlink" href="#content" title="Permalink to this headline">�</a></h2>
+<div class="toctree-wrapper compound">
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="project.html#history">History</a></li>
+<li class="toctree-l2"><a class="reference internal" href="project.html#committers">Committers</a></li>
+<li class="toctree-l2"><a class="reference internal" href="project.html#resources-links">Resources & links</a></li>
+<li class="toctree-l2"><a class="reference internal" href="project.html#roadmap">Roadmap</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="start.html#what-s-next">What’s Next?</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="installation.html#getting-airflow">Getting Airflow</a></li>
+<li class="toctree-l2"><a class="reference internal" href="installation.html#extra-packages">Extra Packages</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#example-pipeline-definition">Example Pipeline definition</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#it-s-a-dag-definition-file">It’s a DAG definition file</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#importing-modules">Importing Modules</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#default-arguments">Default Arguments</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#instantiate-a-dag">Instantiate a DAG</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#tasks">Tasks</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#templating-with-jinja">Templating with Jinja</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#setting-up-dependencies">Setting up Dependencies</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#recap">Recap</a></li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#testing">Testing</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="tutorial.html#running-the-script">Running the Script</a></li>
+<li class="toctree-l3"><a class="reference internal" href="tutorial.html#command-line-metadata-validation">Command Line Metadata Validation</a></li>
+<li class="toctree-l3"><a class="reference internal" href="tutorial.html#id1">Testing</a></li>
+<li class="toctree-l3"><a class="reference internal" href="tutorial.html#backfill">Backfill</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="tutorial.html#what-s-next">What’s Next?</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="configuration.html#setting-configuration-options">Setting Configuration Options</a></li>
+<li class="toctree-l2"><a class="reference internal" href="configuration.html#setting-up-a-backend">Setting up a Backend</a></li>
+<li class="toctree-l2"><a class="reference internal" href="configuration.html#connections">Connections</a></li>
+<li class="toctree-l2"><a class="reference internal" href="configuration.html#scaling-out-with-celery">Scaling Out with Celery</a></li>
+<li class="toctree-l2"><a class="reference internal" href="configuration.html#logs">Logs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="configuration.html#scaling-out-on-mesos-community-contributed">Scaling Out on Mesos (community contributed)</a></li>
+<li class="toctree-l2"><a class="reference internal" href="configuration.html#integration-with-systemd">Integration with systemd</a></li>
+<li class="toctree-l2"><a class="reference internal" href="configuration.html#integration-with-upstart">Integration with upstart</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="ui.html#dags-view">DAGs View</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ui.html#tree-view">Tree View</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ui.html#graph-view">Graph View</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ui.html#variable-view">Variable View</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ui.html#gantt-chart">Gantt Chart</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ui.html#task-duration">Task Duration</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ui.html#code-view">Code View</a></li>
+<li class="toctree-l2"><a class="reference internal" href="ui.html#task-instance-context-menu">Task Instance Context Menu</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="concepts.html#core-ideas">Core Ideas</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#dags">DAGs</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="concepts.html#scope">Scope</a></li>
+<li class="toctree-l4"><a class="reference internal" href="concepts.html#default-arguments">Default Arguments</a></li>
+<li class="toctree-l4"><a class="reference internal" href="concepts.html#context-manager">Context Manager</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#operators">Operators</a><ul>
+<li class="toctree-l4"><a class="reference internal" href="concepts.html#dag-assignment">DAG Assignment</a></li>
+<li class="toctree-l4"><a class="reference internal" href="concepts.html#bitshift-composition">Bitshift Composition</a></li>
+</ul>
+</li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#tasks">Tasks</a></li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#task-instances">Task Instances</a></li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#workflows">Workflows</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="concepts.html#additional-functionality">Additional Functionality</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#hooks">Hooks</a></li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#pools">Pools</a></li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#connections">Connections</a></li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#queues">Queues</a></li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#xcoms">XComs</a></li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#variables">Variables</a></li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#branching">Branching</a></li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#subdags">SubDAGs</a></li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#slas">SLAs</a></li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#trigger-rules">Trigger Rules</a></li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#zombies-undeads">Zombies & Undeads</a></li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#cluster-policy">Cluster Policy</a></li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#documentation-notes">Documentation & Notes</a></li>
+<li class="toctree-l3"><a class="reference internal" href="concepts.html#jinja-templating">Jinja Templating</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="concepts.html#packaged-dags">Packaged dags</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="profiling.html#adhoc-queries">Adhoc Queries</a></li>
+<li class="toctree-l2"><a class="reference internal" href="profiling.html#charts">Charts</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="profiling.html#chart-screenshot">Chart Screenshot</a></li>
+<li class="toctree-l3"><a class="reference internal" href="profiling.html#chart-form-screenshot">Chart Form Screenshot</a></li>
+</ul>
+</li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="scheduler.html#dag-runs">DAG Runs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="scheduler.html#external-triggers">External Triggers</a></li>
+<li class="toctree-l2"><a class="reference internal" href="scheduler.html#to-keep-in-mind">To Keep in Mind</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="plugins.html#what-for">What for?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="plugins.html#why-build-on-top-of-airflow">Why build on top of Airflow?</a></li>
+<li class="toctree-l2"><a class="reference internal" href="plugins.html#interface">Interface</a></li>
+<li class="toctree-l2"><a class="reference internal" href="plugins.html#example">Example</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="security.html#web-authentication">Web Authentication</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="security.html#password">Password</a></li>
+<li class="toctree-l3"><a class="reference internal" href="security.html#ldap">LDAP</a></li>
+<li class="toctree-l3"><a class="reference internal" href="security.html#roll-your-own">Roll your own</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="security.html#multi-tenancy">Multi-tenancy</a></li>
+<li class="toctree-l2"><a class="reference internal" href="security.html#kerberos">Kerberos</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="security.html#limitations">Limitations</a></li>
+<li class="toctree-l3"><a class="reference internal" href="security.html#enabling-kerberos">Enabling kerberos</a></li>
+<li class="toctree-l3"><a class="reference internal" href="security.html#using-kerberos-authentication">Using kerberos authentication</a></li>
+<li class="toctree-l3"><a class="reference internal" href="security.html#github-enterprise-ghe-authentication">GitHub Enterprise (GHE) Authentication</a></li>
+<li class="toctree-l3"><a class="reference internal" href="security.html#setting-up-ghe-authentication">Setting up GHE Authentication</a></li>
+</ul>
+</li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="code.html#operators">Operators</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="code.html#baseoperator">BaseOperator</a></li>
+<li class="toctree-l3"><a class="reference internal" href="code.html#basesensoroperator">BaseSensorOperator</a></li>
+<li class="toctree-l3"><a class="reference internal" href="code.html#module-airflow.operators">Operator API</a></li>
+<li class="toctree-l3"><a class="reference internal" href="code.html#community-contributed-operators">Community-contributed Operators</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="code.html#macros">Macros</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="code.html#default-variables">Default Variables</a></li>
+<li class="toctree-l3"><a class="reference internal" href="code.html#id2">Macros</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="code.html#models">Models</a></li>
+<li class="toctree-l2"><a class="reference internal" href="code.html#module-airflow.hooks">Hooks</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="code.html#community-contributed-hooks">Community contributed hooks</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="code.html#executors">Executors</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="code.html#community-contributed-executors">Community-contributed executors</a></li>
+</ul>
+</li>
+</ul>
+</li>
+</ul>
+</div>
+</div>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="project.html" class="btn btn-neutral float-right" title="Project" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[29/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/airflow/operators/sensors.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/operators/sensors.html b/_modules/airflow/operators/sensors.html
new file mode 100644
index 0000000..b48da21
--- /dev/null
+++ b/_modules/airflow/operators/sensors.html
@@ -0,0 +1,721 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.operators.sensors — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../../../index.html"/>
+ <link rel="up" title="Module code" href="../../index.html"/>
+
+
+ <script src="../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../index.html">Module code</a> »</li>
+
+ <li>airflow.operators.sensors</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.operators.sensors</h1><div class="highlight"><pre>
+<span></span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">print_function</span>
+<span class="kn">from</span> <span class="nn">future</span> <span class="kn">import</span> <span class="n">standard_library</span>
+<span class="n">standard_library</span><span class="o">.</span><span class="n">install_aliases</span><span class="p">()</span>
+<span class="kn">from</span> <span class="nn">builtins</span> <span class="kn">import</span> <span class="nb">str</span>
+<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">from</span> <span class="nn">urllib.parse</span> <span class="kn">import</span> <span class="n">urlparse</span>
+<span class="kn">from</span> <span class="nn">time</span> <span class="kn">import</span> <span class="n">sleep</span>
+
+<span class="kn">from</span> <span class="nn">airflow</span> <span class="kn">import</span> <span class="n">hooks</span><span class="p">,</span> <span class="n">settings</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span><span class="p">,</span> <span class="n">AirflowSensorTimeout</span><span class="p">,</span> <span class="n">AirflowSkipException</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span><span class="p">,</span> <span class="n">TaskInstance</span><span class="p">,</span> <span class="n">Connection</span> <span class="k">as</span> <span class="n">DB</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks</span> <span class="kn">import</span> <span class="n">BaseHook</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.state</span> <span class="kn">import</span> <span class="n">State</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+
+
+<div class="viewcode-block" id="BaseSensorOperator"><a class="viewcode-back" href="../../../code.html#airflow.operators.sensors.BaseSensorOperator">[docs]</a><span class="k">class</span> <span class="nc">BaseSensorOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> Sensor operators are derived from this class an inherit these attributes.</span>
+
+<span class="sd"> Sensor operators keep executing at a time interval and succeed when</span>
+<span class="sd"> a criteria is met and fail if and when they time out.</span>
+
+<span class="sd"> :param soft_fail: Set to true to mark the task as SKIPPED on failure</span>
+<span class="sd"> :type soft_fail: bool</span>
+<span class="sd"> :param poke_interval: Time in seconds that the job should wait in</span>
+<span class="sd"> between each tries</span>
+<span class="sd"> :type poke_interval: int</span>
+<span class="sd"> :param timeout: Time, in seconds before the task times out and fails.</span>
+<span class="sd"> :type timeout: int</span>
+<span class="sd"> '''</span>
+ <span class="n">ui_color</span> <span class="o">=</span> <span class="s1">'#e6f1f2'</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">poke_interval</span><span class="o">=</span><span class="mi">60</span><span class="p">,</span>
+ <span class="n">timeout</span><span class="o">=</span><span class="mi">60</span><span class="o">*</span><span class="mi">60</span><span class="o">*</span><span class="mi">24</span><span class="o">*</span><span class="mi">7</span><span class="p">,</span>
+ <span class="n">soft_fail</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">poke_interval</span> <span class="o">=</span> <span class="n">poke_interval</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">soft_fail</span> <span class="o">=</span> <span class="n">soft_fail</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">timeout</span> <span class="o">=</span> <span class="n">timeout</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="sd">'''</span>
+<span class="sd"> Function that the sensors defined while deriving this class should</span>
+<span class="sd"> override.</span>
+<span class="sd"> '''</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Override me.'</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">started_at</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
+ <span class="k">while</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">poke</span><span class="p">(</span><span class="n">context</span><span class="p">):</span>
+ <span class="n">sleep</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">poke_interval</span><span class="p">)</span>
+ <span class="k">if</span> <span class="p">(</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">-</span> <span class="n">started_at</span><span class="p">)</span><span class="o">.</span><span class="n">seconds</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">timeout</span><span class="p">:</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">soft_fail</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowSkipException</span><span class="p">(</span><span class="s1">'Snap. Time is OUT.'</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowSensorTimeout</span><span class="p">(</span><span class="s1">'Snap. Time is OUT.'</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"Success criteria met. Exiting."</span><span class="p">)</span></div>
+
+
+<span class="k">class</span> <span class="nc">SqlSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Runs a sql statement until a criteria is met. It will keep trying until</span>
+<span class="sd"> sql returns no row, or if the first cell in (0, '0', '').</span>
+
+<span class="sd"> :param conn_id: The connection to run the sensor against</span>
+<span class="sd"> :type conn_id: string</span>
+<span class="sd"> :param sql: The sql to run. To pass, it needs to return at least one cell</span>
+<span class="sd"> that contains a non-zero / empty string value.</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'sql'</span><span class="p">,)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.hql'</span><span class="p">,</span> <span class="s1">'.sql'</span><span class="p">,)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">conn_id</span><span class="p">,</span> <span class="n">sql</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="n">sql</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span> <span class="o">=</span> <span class="n">conn_id</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">BaseHook</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span><span class="p">)</span><span class="o">.</span><span class="n">get_hook</span><span class="p">()</span>
+
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking: '</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+ <span class="n">records</span> <span class="o">=</span> <span class="n">hook</span><span class="o">.</span><span class="n">get_records</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">sql</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">records</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">False</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">if</span> <span class="nb">str</span><span class="p">(</span><span class="n">records</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">])</span> <span class="ow">in</span> <span class="p">(</span><span class="s1">'0'</span><span class="p">,</span> <span class="s1">''</span><span class="p">,):</span>
+ <span class="k">return</span> <span class="bp">False</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">True</span>
+ <span class="k">print</span><span class="p">(</span><span class="n">records</span><span class="p">[</span><span class="mi">0</span><span class="p">][</span><span class="mi">0</span><span class="p">])</span>
+
+
+<span class="k">class</span> <span class="nc">MetastorePartitionSensor</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> An alternative to the HivePartitionSensor that talk directly to the</span>
+<span class="sd"> MySQL db. This was created as a result of observing sub optimal</span>
+<span class="sd"> queries generated by the Metastore thrift service when hitting</span>
+<span class="sd"> subpartitioned tables. The Thrift service's queries were written in a</span>
+<span class="sd"> way that wouldn't leverage the indexes.</span>
+
+<span class="sd"> :param schema: the schema</span>
+<span class="sd"> :type schema: str</span>
+<span class="sd"> :param table: the table</span>
+<span class="sd"> :type table: str</span>
+<span class="sd"> :param partition_name: the partition name, as defined in the PARTITIONS</span>
+<span class="sd"> table of the Metastore. Order of the fields does matter.</span>
+<span class="sd"> Examples: ``ds=2016-01-01`` or</span>
+<span class="sd"> ``ds=2016-01-01/sub=foo`` for a sub partitioned table</span>
+<span class="sd"> :type partition_name: str</span>
+<span class="sd"> :param mysql_conn_id: a reference to the MySQL conn_id for the metastore</span>
+<span class="sd"> :type mysql_conn_id: str</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'partition_name'</span><span class="p">,</span> <span class="s1">'table'</span><span class="p">,</span> <span class="s1">'schema'</span><span class="p">)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">table</span><span class="p">,</span> <span class="n">partition_name</span><span class="p">,</span> <span class="n">schema</span><span class="o">=</span><span class="s2">"default"</span><span class="p">,</span>
+ <span class="n">mysql_conn_id</span><span class="o">=</span><span class="s2">"metastore_mysql"</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">partition_name</span> <span class="o">=</span> <span class="n">partition_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">table</span> <span class="o">=</span> <span class="n">table</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="n">schema</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">first_poke</span> <span class="o">=</span> <span class="bp">True</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">conn_id</span> <span class="o">=</span> <span class="n">mysql_conn_id</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">SqlSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">first_poke</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">first_poke</span> <span class="o">=</span> <span class="bp">False</span>
+ <span class="k">if</span> <span class="s1">'.'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">sql</span> <span class="o">=</span> <span class="s2">"""</span>
+<span class="s2"> SELECT 'X'</span>
+<span class="s2"> FROM PARTITIONS A0</span>
+<span class="s2"> LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID = B0.TBL_ID</span>
+<span class="s2"> LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID</span>
+<span class="s2"> WHERE</span>
+<span class="s2"> B0.TBL_NAME = '{self.table}' AND</span>
+<span class="s2"> C0.NAME = '{self.schema}' AND</span>
+<span class="s2"> A0.PART_NAME = '{self.partition_name}';</span>
+<span class="s2"> """</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">=</span><span class="bp">self</span><span class="p">)</span>
+ <span class="k">return</span> <span class="nb">super</span><span class="p">(</span><span class="n">MetastorePartitionSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">poke</span><span class="p">(</span><span class="n">context</span><span class="p">)</span>
+
+
+<span class="k">class</span> <span class="nc">ExternalTaskSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for a task to complete in a different DAG</span>
+
+<span class="sd"> :param external_dag_id: The dag_id that contains the task you want to</span>
+<span class="sd"> wait for</span>
+<span class="sd"> :type external_dag_id: string</span>
+<span class="sd"> :param external_task_id: The task_id that contains the task you want to</span>
+<span class="sd"> wait for</span>
+<span class="sd"> :type external_task_id: string</span>
+<span class="sd"> :param allowed_states: list of allowed states, default is ``['success']``</span>
+<span class="sd"> :type allowed_states: list</span>
+<span class="sd"> :param execution_delta: time difference with the previous execution to</span>
+<span class="sd"> look at, the default is the same execution_date as the current task.</span>
+<span class="sd"> For yesterday, use [positive!] datetime.timedelta(days=1)</span>
+<span class="sd"> :type execution_delta: datetime.timedelta</span>
+<span class="sd"> """</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">external_dag_id</span><span class="p">,</span>
+ <span class="n">external_task_id</span><span class="p">,</span>
+ <span class="n">allowed_states</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">execution_delta</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">ExternalTaskSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">allowed_states</span> <span class="o">=</span> <span class="n">allowed_states</span> <span class="ow">or</span> <span class="p">[</span><span class="n">State</span><span class="o">.</span><span class="n">SUCCESS</span><span class="p">]</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">execution_delta</span> <span class="o">=</span> <span class="n">execution_delta</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">external_dag_id</span> <span class="o">=</span> <span class="n">external_dag_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">external_task_id</span> <span class="o">=</span> <span class="n">external_task_id</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">execution_delta</span><span class="p">:</span>
+ <span class="n">dttm</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'execution_date'</span><span class="p">]</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">execution_delta</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">dttm</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'execution_date'</span><span class="p">]</span>
+
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Poking for '</span>
+ <span class="s1">'{self.external_dag_id}.'</span>
+ <span class="s1">'{self.external_task_id} on '</span>
+ <span class="s1">'{dttm} ... '</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="n">TI</span> <span class="o">=</span> <span class="n">TaskInstance</span>
+
+ <span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
+ <span class="n">count</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">TI</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span>
+ <span class="n">TI</span><span class="o">.</span><span class="n">dag_id</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">external_dag_id</span><span class="p">,</span>
+ <span class="n">TI</span><span class="o">.</span><span class="n">task_id</span> <span class="o">==</span> <span class="bp">self</span><span class="o">.</span><span class="n">external_task_id</span><span class="p">,</span>
+ <span class="n">TI</span><span class="o">.</span><span class="n">state</span><span class="o">.</span><span class="n">in_</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">allowed_states</span><span class="p">),</span>
+ <span class="n">TI</span><span class="o">.</span><span class="n">execution_date</span> <span class="o">==</span> <span class="n">dttm</span><span class="p">,</span>
+ <span class="p">)</span><span class="o">.</span><span class="n">count</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+ <span class="k">return</span> <span class="n">count</span>
+
+
+<span class="k">class</span> <span class="nc">HivePartitionSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for a partition to show up in Hive</span>
+
+<span class="sd"> :param table: The name of the table to wait for, supports the dot</span>
+<span class="sd"> notation (my_database.my_table)</span>
+<span class="sd"> :type table: string</span>
+<span class="sd"> :param partition: The partition clause to wait for. This is passed as</span>
+<span class="sd"> is to the Metastore Thrift client "get_partitions_by_filter" method,</span>
+<span class="sd"> and apparently supports SQL like notation as in `ds='2015-01-01'</span>
+<span class="sd"> AND type='value'` and > < sings as in "ds>=2015-01-01"</span>
+<span class="sd"> :type partition: string</span>
+<span class="sd"> :param metastore_conn_id: reference to the metastore thrift service</span>
+<span class="sd"> connection id</span>
+<span class="sd"> :type metastore_conn_id: str</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'schema'</span><span class="p">,</span> <span class="s1">'table'</span><span class="p">,</span> <span class="s1">'partition'</span><span class="p">,)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">table</span><span class="p">,</span> <span class="n">partition</span><span class="o">=</span><span class="s2">"ds='{{ ds }}'"</span><span class="p">,</span>
+ <span class="n">metastore_conn_id</span><span class="o">=</span><span class="s1">'metastore_default'</span><span class="p">,</span>
+ <span class="n">schema</span><span class="o">=</span><span class="s1">'default'</span><span class="p">,</span>
+ <span class="n">poke_interval</span><span class="o">=</span><span class="mi">60</span><span class="o">*</span><span class="mi">3</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HivePartitionSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span>
+ <span class="n">poke_interval</span><span class="o">=</span><span class="n">poke_interval</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">partition</span><span class="p">:</span>
+ <span class="n">partition</span> <span class="o">=</span> <span class="s2">"ds='{{ ds }}'"</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">metastore_conn_id</span> <span class="o">=</span> <span class="n">metastore_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">table</span> <span class="o">=</span> <span class="n">table</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">partition</span> <span class="o">=</span> <span class="n">partition</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema</span> <span class="o">=</span> <span class="n">schema</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="k">if</span> <span class="s1">'.'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Poking for table {self.schema}.{self.table}, '</span>
+ <span class="s1">'partition {self.partition}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="s1">'hook'</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">HiveMetastoreHook</span><span class="p">(</span>
+ <span class="n">metastore_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">metastore_conn_id</span><span class="p">)</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">check_for_partition</span><span class="p">(</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">schema</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">table</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">partition</span><span class="p">)</span>
+
+
+<span class="k">class</span> <span class="nc">HdfsSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for a file or folder to land in HDFS</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'filepath'</span><span class="p">,)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">filepath</span><span class="p">,</span>
+ <span class="n">hdfs_conn_id</span><span class="o">=</span><span class="s1">'hdfs_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HdfsSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">filepath</span> <span class="o">=</span> <span class="n">filepath</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">hdfs_conn_id</span> <span class="o">=</span> <span class="n">hdfs_conn_id</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">sb</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">HDFSHook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">hdfs_conn_id</span><span class="p">)</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s2">"snakebite"</span><span class="p">)</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">WARNING</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Poking for file {self.filepath} '</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">files</span> <span class="o">=</span> <span class="p">[</span><span class="n">f</span> <span class="k">for</span> <span class="n">f</span> <span class="ow">in</span> <span class="n">sb</span><span class="o">.</span><span class="n">ls</span><span class="p">([</span><span class="bp">self</span><span class="o">.</span><span class="n">filepath</span><span class="p">])]</span>
+ <span class="k">except</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">False</span>
+ <span class="k">return</span> <span class="bp">True</span>
+
+
+<span class="k">class</span> <span class="nc">WebHdfsSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for a file or folder to land in HDFS</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'filepath'</span><span class="p">,)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">filepath</span><span class="p">,</span>
+ <span class="n">webhdfs_conn_id</span><span class="o">=</span><span class="s1">'webhdfs_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">WebHdfsSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">filepath</span> <span class="o">=</span> <span class="n">filepath</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">webhdfs_conn_id</span> <span class="o">=</span> <span class="n">webhdfs_conn_id</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">c</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">WebHDFSHook</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">webhdfs_conn_id</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Poking for file {self.filepath} '</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="k">return</span> <span class="n">c</span><span class="o">.</span><span class="n">check_for_path</span><span class="p">(</span><span class="n">hdfs_path</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">filepath</span><span class="p">)</span>
+
+
+<span class="k">class</span> <span class="nc">S3KeySensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for a key (a file-like instance on S3) to be present in a S3 bucket.</span>
+<span class="sd"> S3 being a key/value it does not support folders. The path is just a key</span>
+<span class="sd"> a resource.</span>
+
+<span class="sd"> :param bucket_key: The key being waited on. Supports full s3:// style url</span>
+<span class="sd"> or relative path from root level.</span>
+<span class="sd"> :type bucket_key: str</span>
+<span class="sd"> :param bucket_name: Name of the S3 bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> :param wildcard_match: whether the bucket_key should be interpreted as a</span>
+<span class="sd"> Unix wildcard pattern</span>
+<span class="sd"> :type wildcard_match: bool</span>
+<span class="sd"> :param s3_conn_id: a reference to the s3 connection</span>
+<span class="sd"> :type s3_conn_id: str</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'bucket_key'</span><span class="p">,</span> <span class="s1">'bucket_name'</span><span class="p">)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">bucket_key</span><span class="p">,</span>
+ <span class="n">bucket_name</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">wildcard_match</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">s3_conn_id</span><span class="o">=</span><span class="s1">'s3_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">S3KeySensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
+ <span class="n">db</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">DB</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">DB</span><span class="o">.</span><span class="n">conn_id</span> <span class="o">==</span> <span class="n">s3_conn_id</span><span class="p">)</span><span class="o">.</span><span class="n">first</span><span class="p">()</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">db</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"conn_id doesn't exist in the repository"</span><span class="p">)</span>
+ <span class="c1"># Parse</span>
+ <span class="k">if</span> <span class="n">bucket_name</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
+ <span class="n">parsed_url</span> <span class="o">=</span> <span class="n">urlparse</span><span class="p">(</span><span class="n">bucket_key</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">netloc</span> <span class="o">==</span> <span class="s1">''</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Please provide a bucket_name'</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">bucket_name</span> <span class="o">=</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">netloc</span>
+ <span class="k">if</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">path</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'/'</span><span class="p">:</span>
+ <span class="n">bucket_key</span> <span class="o">=</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">path</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">bucket_key</span> <span class="o">=</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">path</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span> <span class="o">=</span> <span class="n">bucket_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bucket_key</span> <span class="o">=</span> <span class="n">bucket_key</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">wildcard_match</span> <span class="o">=</span> <span class="n">wildcard_match</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span> <span class="o">=</span> <span class="n">s3_conn_id</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">S3Hook</span><span class="p">(</span><span class="n">s3_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span><span class="p">)</span>
+ <span class="n">full_url</span> <span class="o">=</span> <span class="s2">"s3://"</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span> <span class="o">+</span> <span class="s2">"/"</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">bucket_key</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for key : {full_url}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">wildcard_match</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">check_for_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">bucket_key</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">check_for_key</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">bucket_key</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span><span class="p">)</span>
+
+
+<span class="k">class</span> <span class="nc">S3PrefixSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for a prefix to exist. A prefix is the first part of a key,</span>
+<span class="sd"> thus enabling checking of constructs similar to glob airfl* or</span>
+<span class="sd"> SQL LIKE 'airfl%'. There is the possibility to precise a delimiter to</span>
+<span class="sd"> indicate the hierarchy or keys, meaning that the match will stop at that</span>
+<span class="sd"> delimiter. Current code accepts sane delimiters, i.e. characters that</span>
+<span class="sd"> are NOT special characters in the Python regex engine.</span>
+
+<span class="sd"> :param bucket_name: Name of the S3 bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> :param prefix: The prefix being waited on. Relative path from bucket root level.</span>
+<span class="sd"> :type prefix: str</span>
+<span class="sd"> :param delimiter: The delimiter intended to show hierarchy.</span>
+<span class="sd"> Defaults to '/'.</span>
+<span class="sd"> :type delimiter: str</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'prefix'</span><span class="p">,</span> <span class="s1">'bucket_name'</span><span class="p">)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span>
+ <span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">'/'</span><span class="p">,</span>
+ <span class="n">s3_conn_id</span><span class="o">=</span><span class="s1">'s3_default'</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">S3PrefixSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="n">session</span> <span class="o">=</span> <span class="n">settings</span><span class="o">.</span><span class="n">Session</span><span class="p">()</span>
+ <span class="n">db</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">DB</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="n">DB</span><span class="o">.</span><span class="n">conn_id</span> <span class="o">==</span> <span class="n">s3_conn_id</span><span class="p">)</span><span class="o">.</span><span class="n">first</span><span class="p">()</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">db</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"conn_id doesn't exist in the repository"</span><span class="p">)</span>
+ <span class="c1"># Parse</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span> <span class="o">=</span> <span class="n">bucket_name</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">prefix</span> <span class="o">=</span> <span class="n">prefix</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span> <span class="o">=</span> <span class="n">delimiter</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">full_url</span> <span class="o">=</span> <span class="s2">"s3://"</span> <span class="o">+</span> <span class="n">bucket_name</span> <span class="o">+</span> <span class="s1">'/'</span> <span class="o">+</span> <span class="n">prefix</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span> <span class="o">=</span> <span class="n">s3_conn_id</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
+ <span class="n">session</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking for prefix : {self.prefix}</span><span class="se">\n</span><span class="s1">'</span>
+ <span class="s1">'in bucket s3://{self.bucket_name}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="n">hook</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">S3Hook</span><span class="p">(</span><span class="n">s3_conn_id</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">hook</span><span class="o">.</span><span class="n">check_for_prefix</span><span class="p">(</span>
+ <span class="n">prefix</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">prefix</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">delimiter</span><span class="p">,</span>
+ <span class="n">bucket_name</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">bucket_name</span><span class="p">)</span>
+
+
+<span class="k">class</span> <span class="nc">TimeSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits until the specified time of the day.</span>
+
+<span class="sd"> :param target_time: time after which the job succeeds</span>
+<span class="sd"> :type target_time: datetime.time</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">()</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">target_time</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">TimeSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">target_time</span> <span class="o">=</span> <span class="n">target_time</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span>
+ <span class="s1">'Checking if the time ({0}) has come'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">target_time</span><span class="p">))</span>
+ <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span><span class="o">.</span><span class="n">time</span><span class="p">()</span> <span class="o">></span> <span class="bp">self</span><span class="o">.</span><span class="n">target_time</span>
+
+
+<span class="k">class</span> <span class="nc">TimeDeltaSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Waits for a timedelta after the task's execution_date + schedule_interval.</span>
+<span class="sd"> In Airflow, the daily task stamped with ``execution_date``</span>
+<span class="sd"> 2016-01-01 can only start running on 2016-01-02. The timedelta here</span>
+<span class="sd"> represents the time after the execution period has closed.</span>
+
+<span class="sd"> :param delta: time length to wait after execution_date before succeeding</span>
+<span class="sd"> :type delta: datetime.timedelta</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="nb">tuple</span><span class="p">()</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">delta</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">TimeDeltaSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">delta</span> <span class="o">=</span> <span class="n">delta</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">dag</span> <span class="o">=</span> <span class="n">context</span><span class="p">[</span><span class="s1">'dag'</span><span class="p">]</span>
+ <span class="n">target_dttm</span> <span class="o">=</span> <span class="n">dag</span><span class="o">.</span><span class="n">following_schedule</span><span class="p">(</span><span class="n">context</span><span class="p">[</span><span class="s1">'execution_date'</span><span class="p">])</span>
+ <span class="n">target_dttm</span> <span class="o">+=</span> <span class="bp">self</span><span class="o">.</span><span class="n">delta</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Checking if the time ({0}) has come'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">target_dttm</span><span class="p">))</span>
+ <span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span> <span class="o">></span> <span class="n">target_dttm</span>
+
+
+<span class="k">class</span> <span class="nc">HttpSensor</span><span class="p">(</span><span class="n">BaseSensorOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Executes a HTTP get statement and returns False on failure:</span>
+<span class="sd"> 404 not found or response_check function returned False</span>
+
+<span class="sd"> :param http_conn_id: The connection to run the sensor against</span>
+<span class="sd"> :type http_conn_id: string</span>
+<span class="sd"> :param endpoint: The relative part of the full url</span>
+<span class="sd"> :type endpoint: string</span>
+<span class="sd"> :param params: The parameters to be added to the GET url</span>
+<span class="sd"> :type params: a dictionary of string key/value pairs</span>
+<span class="sd"> :param headers: The HTTP headers to be added to the GET request</span>
+<span class="sd"> :type headers: a dictionary of string key/value pairs</span>
+<span class="sd"> :param response_check: A check against the 'requests' response object.</span>
+<span class="sd"> Returns True for 'pass' and False otherwise.</span>
+<span class="sd"> :type response_check: A lambda or defined function.</span>
+<span class="sd"> :param extra_options: Extra options for the 'requests' library, see the</span>
+<span class="sd"> 'requests' documentation (options to modify timeout, ssl, etc.)</span>
+<span class="sd"> :type extra_options: A dictionary of options, where key is string and value</span>
+<span class="sd"> depends on the option that's being modified.</span>
+<span class="sd"> """</span>
+
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'endpoint'</span><span class="p">,)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">endpoint</span><span class="p">,</span>
+ <span class="n">http_conn_id</span><span class="o">=</span><span class="s1">'http_default'</span><span class="p">,</span>
+ <span class="n">params</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">headers</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">response_check</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">extra_options</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+ <span class="nb">super</span><span class="p">(</span><span class="n">HttpSensor</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span> <span class="o">=</span> <span class="n">endpoint</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">http_conn_id</span> <span class="o">=</span> <span class="n">http_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">params</span> <span class="o">=</span> <span class="n">params</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">headers</span> <span class="o">=</span> <span class="n">headers</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">extra_options</span> <span class="o">=</span> <span class="n">extra_options</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">response_check</span> <span class="o">=</span> <span class="n">response_check</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">hook</span> <span class="o">=</span> <span class="n">hooks</span><span class="o">.</span><span class="n">HttpHook</span><span class="p">(</span><span class="n">method</span><span class="o">=</span><span class="s1">'GET'</span><span class="p">,</span> <span class="n">http_conn_id</span><span class="o">=</span><span class="n">http_conn_id</span><span class="p">)</span>
+
+ <span class="k">def</span> <span class="nf">poke</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Poking: '</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span><span class="p">)</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">response</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">hook</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">endpoint</span><span class="p">,</span>
+ <span class="n">data</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">params</span><span class="p">,</span>
+ <span class="n">headers</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">headers</span><span class="p">,</span>
+ <span class="n">extra_options</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">extra_options</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">response_check</span><span class="p">:</span>
+ <span class="c1"># run content check on response</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">response_check</span><span class="p">(</span><span class="n">response</span><span class="p">)</span>
+ <span class="k">except</span> <span class="n">AirflowException</span> <span class="k">as</span> <span class="n">ae</span><span class="p">:</span>
+ <span class="k">if</span> <span class="n">ae</span><span class="o">.</span><span class="n">message</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">"404"</span><span class="p">):</span>
+ <span class="k">return</span> <span class="bp">False</span>
+
+ <span class="k">raise</span> <span class="n">ae</span>
+
+ <span class="k">return</span> <span class="bp">True</span>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[30/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/airflow/operators/docker_operator.html
----------------------------------------------------------------------
diff --git a/_modules/airflow/operators/docker_operator.html b/_modules/airflow/operators/docker_operator.html
new file mode 100644
index 0000000..a1d8fca
--- /dev/null
+++ b/_modules/airflow/operators/docker_operator.html
@@ -0,0 +1,383 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>airflow.operators.docker_operator — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../../../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../../../index.html"/>
+ <link rel="up" title="Module code" href="../../index.html"/>
+
+
+ <script src="../../../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../../../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../../../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../../../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../../../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../../../index.html">Docs</a> »</li>
+
+ <li><a href="../../index.html">Module code</a> »</li>
+
+ <li>airflow.operators.docker_operator</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for airflow.operators.docker_operator</h1><div class="highlight"><pre>
+<span></span><span class="kn">import</span> <span class="nn">json</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="kn">import</span> <span class="n">BaseOperator</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.decorators</span> <span class="kn">import</span> <span class="n">apply_defaults</span>
+<span class="kn">from</span> <span class="nn">airflow.utils.file</span> <span class="kn">import</span> <span class="n">TemporaryDirectory</span>
+<span class="kn">from</span> <span class="nn">docker</span> <span class="kn">import</span> <span class="n">Client</span><span class="p">,</span> <span class="n">tls</span>
+<span class="kn">import</span> <span class="nn">ast</span>
+
+
+<div class="viewcode-block" id="DockerOperator"><a class="viewcode-back" href="../../../code.html#airflow.operators.docker_operator.DockerOperator">[docs]</a><span class="k">class</span> <span class="nc">DockerOperator</span><span class="p">(</span><span class="n">BaseOperator</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Execute a command inside a docker container.</span>
+
+<span class="sd"> A temporary directory is created on the host and mounted into a container to allow storing files</span>
+<span class="sd"> that together exceed the default disk size of 10GB in a container. The path to the mounted</span>
+<span class="sd"> directory can be accessed via the environment variable ``AIRFLOW_TMP_DIR``.</span>
+
+<span class="sd"> :param image: Docker image from which to create the container.</span>
+<span class="sd"> :type image: str</span>
+<span class="sd"> :param api_version: Remote API version.</span>
+<span class="sd"> :type api_version: str</span>
+<span class="sd"> :param command: Command to be run in the container.</span>
+<span class="sd"> :type command: str or list</span>
+<span class="sd"> :param cpus: Number of CPUs to assign to the container.</span>
+<span class="sd"> This value gets multiplied with 1024. See</span>
+<span class="sd"> https://docs.docker.com/engine/reference/run/#cpu-share-constraint</span>
+<span class="sd"> :type cpus: float</span>
+<span class="sd"> :param docker_url: URL of the host running the docker daemon.</span>
+<span class="sd"> :type docker_url: str</span>
+<span class="sd"> :param environment: Environment variables to set in the container.</span>
+<span class="sd"> :type environment: dict</span>
+<span class="sd"> :param force_pull: Pull the docker image on every run.</span>
+<span class="sd"> :type force_pull: bool</span>
+<span class="sd"> :param mem_limit: Maximum amount of memory the container can use. Either a float value, which</span>
+<span class="sd"> represents the limit in bytes, or a string like ``128m`` or ``1g``.</span>
+<span class="sd"> :type mem_limit: float or str</span>
+<span class="sd"> :param network_mode: Network mode for the container.</span>
+<span class="sd"> :type network_mode: str</span>
+<span class="sd"> :param tls_ca_cert: Path to a PEM-encoded certificate authority to secure the docker connection.</span>
+<span class="sd"> :type tls_ca_cert: str</span>
+<span class="sd"> :param tls_client_cert: Path to the PEM-encoded certificate used to authenticate docker client.</span>
+<span class="sd"> :type tls_client_cert: str</span>
+<span class="sd"> :param tls_client_key: Path to the PEM-encoded key used to authenticate docker client.</span>
+<span class="sd"> :type tls_client_key: str</span>
+<span class="sd"> :param tls_hostname: Hostname to match against the docker server certificate or False to</span>
+<span class="sd"> disable the check.</span>
+<span class="sd"> :type tls_hostname: str or bool</span>
+<span class="sd"> :param tls_ssl_version: Version of SSL to use when communicating with docker daemon.</span>
+<span class="sd"> :type tls_ssl_version: str</span>
+<span class="sd"> :param tmp_dir: Mount point inside the container to a temporary directory created on the host by</span>
+<span class="sd"> the operator. The path is also made available via the environment variable</span>
+<span class="sd"> ``AIRFLOW_TMP_DIR`` inside the container.</span>
+<span class="sd"> :type tmp_dir: str</span>
+<span class="sd"> :param user: Default user inside the docker container.</span>
+<span class="sd"> :type user: int or str</span>
+<span class="sd"> :param volumes: List of volumes to mount into the container, e.g.</span>
+<span class="sd"> ``['/host/path:/container/path', '/host/path2:/container/path2:ro']``.</span>
+<span class="sd"> :param xcom_push: Does the stdout will be pushed to the next step using XCom.</span>
+<span class="sd"> The default is False.</span>
+<span class="sd"> :type xcom_push: bool</span>
+<span class="sd"> :param xcom_all: Push all the stdout or just the last line. The default is False (last line).</span>
+<span class="sd"> :type xcom_all: bool</span>
+<span class="sd"> """</span>
+ <span class="n">template_fields</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'command'</span><span class="p">,)</span>
+ <span class="n">template_ext</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'.sh'</span><span class="p">,</span> <span class="s1">'.bash'</span><span class="p">,)</span>
+
+ <span class="nd">@apply_defaults</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">image</span><span class="p">,</span>
+ <span class="n">api_version</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">command</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">cpus</span><span class="o">=</span><span class="mf">1.0</span><span class="p">,</span>
+ <span class="n">docker_url</span><span class="o">=</span><span class="s1">'unix://var/run/docker.sock'</span><span class="p">,</span>
+ <span class="n">environment</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">force_pull</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">mem_limit</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">network_mode</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">tls_ca_cert</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">tls_client_cert</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">tls_client_key</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">tls_hostname</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">tls_ssl_version</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">tmp_dir</span><span class="o">=</span><span class="s1">'/tmp/airflow'</span><span class="p">,</span>
+ <span class="n">user</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">volumes</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">xcom_push</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">xcom_all</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="o">*</span><span class="n">args</span><span class="p">,</span>
+ <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
+
+ <span class="nb">super</span><span class="p">(</span><span class="n">DockerOperator</span><span class="p">,</span> <span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">api_version</span> <span class="o">=</span> <span class="n">api_version</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">command</span> <span class="o">=</span> <span class="n">command</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cpus</span> <span class="o">=</span> <span class="n">cpus</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">docker_url</span> <span class="o">=</span> <span class="n">docker_url</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">environment</span> <span class="o">=</span> <span class="n">environment</span> <span class="ow">or</span> <span class="p">{}</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">force_pull</span> <span class="o">=</span> <span class="n">force_pull</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">image</span> <span class="o">=</span> <span class="n">image</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">mem_limit</span> <span class="o">=</span> <span class="n">mem_limit</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">network_mode</span> <span class="o">=</span> <span class="n">network_mode</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">tls_ca_cert</span> <span class="o">=</span> <span class="n">tls_ca_cert</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">tls_client_cert</span> <span class="o">=</span> <span class="n">tls_client_cert</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">tls_client_key</span> <span class="o">=</span> <span class="n">tls_client_key</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">tls_hostname</span> <span class="o">=</span> <span class="n">tls_hostname</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">tls_ssl_version</span> <span class="o">=</span> <span class="n">tls_ssl_version</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">tmp_dir</span> <span class="o">=</span> <span class="n">tmp_dir</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">user</span> <span class="o">=</span> <span class="n">user</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">volumes</span> <span class="o">=</span> <span class="n">volumes</span> <span class="ow">or</span> <span class="p">[]</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">xcom_push</span> <span class="o">=</span> <span class="n">xcom_push</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">xcom_all</span> <span class="o">=</span> <span class="n">xcom_all</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">cli</span> <span class="o">=</span> <span class="bp">None</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">container</span> <span class="o">=</span> <span class="bp">None</span>
+
+ <span class="k">def</span> <span class="nf">execute</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">context</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Starting docker container from image '</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">image</span><span class="p">)</span>
+
+ <span class="n">tls_config</span> <span class="o">=</span> <span class="bp">None</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">tls_ca_cert</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">tls_client_cert</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">tls_client_key</span><span class="p">:</span>
+ <span class="n">tls_config</span> <span class="o">=</span> <span class="n">tls</span><span class="o">.</span><span class="n">TLSConfig</span><span class="p">(</span>
+ <span class="n">ca_cert</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">tls_ca_cert</span><span class="p">,</span>
+ <span class="n">client_cert</span><span class="o">=</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">tls_client_cert</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">tls_client_key</span><span class="p">),</span>
+ <span class="n">verify</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
+ <span class="n">ssl_version</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">tls_ssl_version</span><span class="p">,</span>
+ <span class="n">assert_hostname</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">tls_hostname</span>
+ <span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">docker_url</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">docker_url</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s1">'tcp://'</span><span class="p">,</span> <span class="s1">'https://'</span><span class="p">)</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">cli</span> <span class="o">=</span> <span class="n">Client</span><span class="p">(</span><span class="n">base_url</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">docker_url</span><span class="p">,</span> <span class="n">version</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">api_version</span><span class="p">,</span> <span class="n">tls</span><span class="o">=</span><span class="n">tls_config</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="s1">':'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">image</span><span class="p">:</span>
+ <span class="n">image</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">image</span> <span class="o">+</span> <span class="s1">':latest'</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">image</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">image</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">force_pull</span> <span class="ow">or</span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">images</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">image</span><span class="p">))</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Pulling docker image '</span> <span class="o">+</span> <span class="n">image</span><span class="p">)</span>
+ <span class="k">for</span> <span class="n">l</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">pull</span><span class="p">(</span><span class="n">image</span><span class="p">,</span> <span class="n">stream</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span>
+ <span class="n">output</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="n">l</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"{}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">output</span><span class="p">[</span><span class="s1">'status'</span><span class="p">]))</span>
+
+ <span class="n">cpu_shares</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="nb">round</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">cpus</span> <span class="o">*</span> <span class="mi">1024</span><span class="p">))</span>
+
+ <span class="k">with</span> <span class="n">TemporaryDirectory</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="s1">'airflowtmp'</span><span class="p">)</span> <span class="k">as</span> <span class="n">host_tmp_dir</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">environment</span><span class="p">[</span><span class="s1">'AIRFLOW_TMP_DIR'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">tmp_dir</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">volumes</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s1">'{0}:{1}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">host_tmp_dir</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">tmp_dir</span><span class="p">))</span>
+
+ <span class="bp">self</span><span class="o">.</span><span class="n">container</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">create_container</span><span class="p">(</span>
+ <span class="n">command</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">get_command</span><span class="p">(),</span>
+ <span class="n">cpu_shares</span><span class="o">=</span><span class="n">cpu_shares</span><span class="p">,</span>
+ <span class="n">environment</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">environment</span><span class="p">,</span>
+ <span class="n">host_config</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">create_host_config</span><span class="p">(</span><span class="n">binds</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">volumes</span><span class="p">,</span>
+ <span class="n">network_mode</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">network_mode</span><span class="p">),</span>
+ <span class="n">image</span><span class="o">=</span><span class="n">image</span><span class="p">,</span>
+ <span class="n">mem_limit</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">mem_limit</span><span class="p">,</span>
+ <span class="n">user</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">user</span>
+ <span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">start</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">container</span><span class="p">[</span><span class="s1">'Id'</span><span class="p">])</span>
+
+ <span class="n">line</span> <span class="o">=</span> <span class="s1">''</span>
+ <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">logs</span><span class="p">(</span><span class="n">container</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">container</span><span class="p">[</span><span class="s1">'Id'</span><span class="p">],</span> <span class="n">stream</span><span class="o">=</span><span class="bp">True</span><span class="p">):</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"{}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">()))</span>
+
+ <span class="n">exit_code</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">wait</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">container</span><span class="p">[</span><span class="s1">'Id'</span><span class="p">])</span>
+ <span class="k">if</span> <span class="n">exit_code</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'docker container failed'</span><span class="p">)</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">xcom_push</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">logs</span><span class="p">(</span><span class="n">container</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">container</span><span class="p">[</span><span class="s1">'Id'</span><span class="p">])</span> <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">xcom_all</span> <span class="k">else</span> <span class="nb">str</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">())</span>
+
+ <span class="k">def</span> <span class="nf">get_command</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">command</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">command</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="s1">'['</span><span class="p">)</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
+ <span class="n">commands</span> <span class="o">=</span> <span class="n">ast</span><span class="o">.</span><span class="n">literal_eval</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">command</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">commands</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">command</span>
+ <span class="k">return</span> <span class="n">commands</span>
+
+ <span class="k">def</span> <span class="nf">on_kill</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">cli</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Stopping docker container'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">stop</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">container</span><span class="p">[</span><span class="s1">'Id'</span><span class="p">])</span></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../../../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../../../_static/jquery.js"></script>
+ <script type="text/javascript" src="../../../_static/underscore.js"></script>
+ <script type="text/javascript" src="../../../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../../../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
[34/34] incubator-airflow-site git commit: Initial commit
Posted by ma...@apache.org.
Initial commit
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/commit/9e19165c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/tree/9e19165c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/diff/9e19165c
Branch: refs/heads/asf-site
Commit: 9e19165cac0531b3f00dff4281c9910620b824c6
Parents:
Author: Maxime Beauchemin <ma...@gmail.com>
Authored: Fri Jun 3 11:44:05 2016 -0700
Committer: Maxime Beauchemin <ma...@gmail.com>
Committed: Fri Jun 3 11:44:05 2016 -0700
----------------------------------------------------------------------
_images/adhoc.png | Bin 0 -> 182773 bytes
_images/airflow.gif | Bin 0 -> 622963 bytes
_images/apache.jpg | Bin 0 -> 43364 bytes
_images/branch_bad.png | Bin 0 -> 8825 bytes
_images/branch_good.png | Bin 0 -> 12035 bytes
_images/chart.png | Bin 0 -> 169382 bytes
_images/chart_form.png | Bin 0 -> 203224 bytes
_images/code.png | Bin 0 -> 218215 bytes
_images/connections.png | Bin 0 -> 93057 bytes
_images/context.png | Bin 0 -> 238360 bytes
_images/dags.png | Bin 0 -> 115068 bytes
_images/duration.png | Bin 0 -> 254497 bytes
_images/gantt.png | Bin 0 -> 213845 bytes
_images/graph.png | Bin 0 -> 333294 bytes
_images/incubator.jpg | Bin 0 -> 91227 bytes
_images/pin_large.png | Bin 0 -> 358276 bytes
_images/subdag_after.png | Bin 0 -> 30245 bytes
_images/subdag_before.png | Bin 0 -> 70382 bytes
_images/subdag_zoom.png | Bin 0 -> 150185 bytes
_images/tree.png | Bin 0 -> 163147 bytes
_images/variable_hidden.png | Bin 0 -> 154299 bytes
_modules/S3_hook.html | 604 +
.../contrib/operators/hipchat_operator.html | 330 +
_modules/airflow/executors/celery_executor.html | 298 +
_modules/airflow/executors/local_executor.html | 276 +
.../airflow/executors/sequential_executor.html | 238 +
_modules/airflow/macros.html | 255 +
_modules/airflow/macros/hive.html | 298 +
_modules/airflow/models.html | 3802 ++++++
_modules/airflow/operators/docker_operator.html | 383 +
_modules/airflow/operators/sensors.html | 721 ++
_modules/bash_operator.html | 290 +
_modules/cloudant_hook.html | 274 +
_modules/dagrun_operator.html | 260 +
_modules/dbapi_hook.html | 426 +
_modules/druid_hook.html | 369 +
_modules/dummy_operator.html | 219 +
_modules/email_operator.html | 240 +
_modules/ftp_hook.html | 427 +
_modules/gcs_hook.html | 296 +
_modules/generic_transfer.html | 264 +
_modules/hive_hooks.html | 743 ++
_modules/hive_operator.html | 272 +
_modules/hive_to_druid.html | 316 +
_modules/hive_to_mysql.html | 294 +
_modules/hive_to_samba_operator.html | 246 +
_modules/http_hook.html | 310 +
_modules/http_operator.html | 265 +
_modules/index.html | 245 +
_modules/mssql_hook.html | 228 +
_modules/mssql_operator.html | 234 +
_modules/mssql_to_hive.html | 312 +
_modules/mysql_hook.html | 267 +
_modules/mysql_operator.html | 240 +
_modules/mysql_to_hive.html | 316 +
_modules/postgres_hook.html | 236 +
_modules/postgres_operator.html | 239 +
_modules/presto_check_operator.html | 303 +
_modules/presto_hook.html | 298 +
_modules/python_operator.html | 338 +
_modules/s3_to_hive_operator.html | 353 +
_modules/sensors.html | 721 ++
_modules/slack_operator.html | 304 +
_modules/sqlite_hook.html | 222 +
_modules/ssh_execute_operator.html | 343 +
_modules/ssh_hook.html | 353 +
_modules/vertica_hook.html | 247 +
_modules/vertica_operator.html | 233 +
_modules/vertica_to_hive.html | 316 +
_modules/webhdfs_hook.html | 287 +
_sources/cli.txt | 11 +
_sources/code.txt | 243 +
_sources/concepts.txt | 758 ++
_sources/configuration.txt | 230 +
_sources/faq.txt | 100 +
_sources/index.txt | 75 +
_sources/installation.txt | 90 +
_sources/license.txt | 211 +
_sources/plugins.txt | 139 +
_sources/profiling.txt | 39 +
_sources/project.txt | 58 +
_sources/scheduler.txt | 101 +
_sources/security.txt | 249 +
_sources/start.txt | 49 +
_sources/tutorial.txt | 429 +
_sources/ui.txt | 102 +
_static/ajax-loader.gif | Bin 0 -> 673 bytes
_static/apache.jpg | Bin 0 -> 43364 bytes
_static/basic.css | 608 +
_static/comment-bright.png | Bin 0 -> 3500 bytes
_static/comment-close.png | Bin 0 -> 3578 bytes
_static/comment.png | Bin 0 -> 3445 bytes
_static/css/badge_only.css | 2 +
_static/css/theme.css | 5 +
_static/doctools.js | 287 +
_static/down-pressed.png | Bin 0 -> 347 bytes
_static/down.png | Bin 0 -> 347 bytes
_static/file.png | Bin 0 -> 358 bytes
_static/fonts/Inconsolata-Bold.ttf | Bin 0 -> 66352 bytes
_static/fonts/Inconsolata-Regular.ttf | Bin 0 -> 84548 bytes
_static/fonts/Lato-Bold.ttf | Bin 0 -> 121788 bytes
_static/fonts/Lato-Regular.ttf | Bin 0 -> 120196 bytes
_static/fonts/RobotoSlab-Bold.ttf | Bin 0 -> 170616 bytes
_static/fonts/RobotoSlab-Regular.ttf | Bin 0 -> 169064 bytes
_static/fonts/fontawesome-webfont.eot | Bin 0 -> 56006 bytes
_static/fonts/fontawesome-webfont.svg | 520 +
_static/fonts/fontawesome-webfont.ttf | Bin 0 -> 112160 bytes
_static/fonts/fontawesome-webfont.woff | Bin 0 -> 65452 bytes
_static/incubator.jpg | Bin 0 -> 91227 bytes
_static/jquery-1.11.1.js | 10308 +++++++++++++++++
_static/jquery.js | 4 +
_static/js/modernizr.min.js | 4 +
_static/js/theme.js | 153 +
_static/minus.png | Bin 0 -> 173 bytes
_static/plus.png | Bin 0 -> 173 bytes
_static/pygments.css | 65 +
_static/searchtools.js | 651 ++
_static/underscore-1.3.1.js | 999 ++
_static/underscore.js | 31 +
_static/up-pressed.png | Bin 0 -> 345 bytes
_static/up.png | Bin 0 -> 345 bytes
_static/websupport.js | 808 ++
cli.html | 1035 ++
code.html | 3517 ++++++
concepts.html | 897 ++
configuration.html | 419 +
faq.html | 293 +
genindex.html | 1258 ++
index.html | 417 +
installation.html | 358 +
license.html | 418 +
objects.inv | Bin 0 -> 2103 bytes
plugins.html | 343 +
profiling.html | 250 +
project.html | 268 +
py-modindex.html | 262 +
scheduler.html | 328 +
search.html | 214 +
searchindex.js | 1 +
security.html | 436 +
start.html | 256 +
tutorial.html | 622 +
ui.html | 296 +
143 files changed, 48568 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/adhoc.png
----------------------------------------------------------------------
diff --git a/_images/adhoc.png b/_images/adhoc.png
new file mode 100644
index 0000000..77ea780
Binary files /dev/null and b/_images/adhoc.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/airflow.gif
----------------------------------------------------------------------
diff --git a/_images/airflow.gif b/_images/airflow.gif
new file mode 100644
index 0000000..1889b86
Binary files /dev/null and b/_images/airflow.gif differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/apache.jpg
----------------------------------------------------------------------
diff --git a/_images/apache.jpg b/_images/apache.jpg
new file mode 100644
index 0000000..312251f
Binary files /dev/null and b/_images/apache.jpg differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/branch_bad.png
----------------------------------------------------------------------
diff --git a/_images/branch_bad.png b/_images/branch_bad.png
new file mode 100644
index 0000000..586844f
Binary files /dev/null and b/_images/branch_bad.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/branch_good.png
----------------------------------------------------------------------
diff --git a/_images/branch_good.png b/_images/branch_good.png
new file mode 100644
index 0000000..fbd4650
Binary files /dev/null and b/_images/branch_good.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/chart.png
----------------------------------------------------------------------
diff --git a/_images/chart.png b/_images/chart.png
new file mode 100644
index 0000000..bfca26b
Binary files /dev/null and b/_images/chart.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/chart_form.png
----------------------------------------------------------------------
diff --git a/_images/chart_form.png b/_images/chart_form.png
new file mode 100644
index 0000000..f73daf5
Binary files /dev/null and b/_images/chart_form.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/code.png
----------------------------------------------------------------------
diff --git a/_images/code.png b/_images/code.png
new file mode 100644
index 0000000..ac49291
Binary files /dev/null and b/_images/code.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/connections.png
----------------------------------------------------------------------
diff --git a/_images/connections.png b/_images/connections.png
new file mode 100644
index 0000000..d07a130
Binary files /dev/null and b/_images/connections.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/context.png
----------------------------------------------------------------------
diff --git a/_images/context.png b/_images/context.png
new file mode 100644
index 0000000..de75e48
Binary files /dev/null and b/_images/context.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/dags.png
----------------------------------------------------------------------
diff --git a/_images/dags.png b/_images/dags.png
new file mode 100644
index 0000000..a551f02
Binary files /dev/null and b/_images/dags.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/duration.png
----------------------------------------------------------------------
diff --git a/_images/duration.png b/_images/duration.png
new file mode 100644
index 0000000..18d723c
Binary files /dev/null and b/_images/duration.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/gantt.png
----------------------------------------------------------------------
diff --git a/_images/gantt.png b/_images/gantt.png
new file mode 100644
index 0000000..c462adb
Binary files /dev/null and b/_images/gantt.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/graph.png
----------------------------------------------------------------------
diff --git a/_images/graph.png b/_images/graph.png
new file mode 100644
index 0000000..cbc58e6
Binary files /dev/null and b/_images/graph.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/incubator.jpg
----------------------------------------------------------------------
diff --git a/_images/incubator.jpg b/_images/incubator.jpg
new file mode 100644
index 0000000..6f34a85
Binary files /dev/null and b/_images/incubator.jpg differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/pin_large.png
----------------------------------------------------------------------
diff --git a/_images/pin_large.png b/_images/pin_large.png
new file mode 100644
index 0000000..986c88b
Binary files /dev/null and b/_images/pin_large.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/subdag_after.png
----------------------------------------------------------------------
diff --git a/_images/subdag_after.png b/_images/subdag_after.png
new file mode 100644
index 0000000..166a6de
Binary files /dev/null and b/_images/subdag_after.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/subdag_before.png
----------------------------------------------------------------------
diff --git a/_images/subdag_before.png b/_images/subdag_before.png
new file mode 100644
index 0000000..ebc3e58
Binary files /dev/null and b/_images/subdag_before.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/subdag_zoom.png
----------------------------------------------------------------------
diff --git a/_images/subdag_zoom.png b/_images/subdag_zoom.png
new file mode 100644
index 0000000..08fcf5c
Binary files /dev/null and b/_images/subdag_zoom.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/tree.png
----------------------------------------------------------------------
diff --git a/_images/tree.png b/_images/tree.png
new file mode 100644
index 0000000..f3796b0
Binary files /dev/null and b/_images/tree.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_images/variable_hidden.png
----------------------------------------------------------------------
diff --git a/_images/variable_hidden.png b/_images/variable_hidden.png
new file mode 100644
index 0000000..e081ca3
Binary files /dev/null and b/_images/variable_hidden.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/_modules/S3_hook.html
----------------------------------------------------------------------
diff --git a/_modules/S3_hook.html b/_modules/S3_hook.html
new file mode 100644
index 0000000..e18ec5f
--- /dev/null
+++ b/_modules/S3_hook.html
@@ -0,0 +1,604 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>S3_hook — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="../index.html"/>
+ <link rel="up" title="Module code" href="index.html"/>
+
+
+ <script src="../_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="../index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="../index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="../index.html">Docs</a> »</li>
+
+ <li><a href="index.html">Module code</a> »</li>
+
+ <li>S3_hook</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <h1>Source code for S3_hook</h1><div class="highlight"><pre>
+<span></span><span class="c1"># -*- coding: utf-8 -*-</span>
+<span class="c1">#</span>
+<span class="c1"># Licensed under the Apache License, Version 2.0 (the "License");</span>
+<span class="c1"># you may not use this file except in compliance with the License.</span>
+<span class="c1"># You may obtain a copy of the License at</span>
+<span class="c1">#</span>
+<span class="c1"># http://www.apache.org/licenses/LICENSE-2.0</span>
+<span class="c1">#</span>
+<span class="c1"># Unless required by applicable law or agreed to in writing, software</span>
+<span class="c1"># distributed under the License is distributed on an "AS IS" BASIS,</span>
+<span class="c1"># WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.</span>
+<span class="c1"># See the License for the specific language governing permissions and</span>
+<span class="c1"># limitations under the License.</span>
+
+<span class="kn">from</span> <span class="nn">future</span> <span class="kn">import</span> <span class="n">standard_library</span>
+<span class="n">standard_library</span><span class="o">.</span><span class="n">install_aliases</span><span class="p">()</span>
+<span class="kn">import</span> <span class="nn">logging</span>
+<span class="kn">import</span> <span class="nn">re</span>
+<span class="kn">import</span> <span class="nn">fnmatch</span>
+<span class="kn">import</span> <span class="nn">configparser</span>
+<span class="kn">import</span> <span class="nn">math</span>
+<span class="kn">import</span> <span class="nn">os</span>
+<span class="kn">from</span> <span class="nn">urllib.parse</span> <span class="kn">import</span> <span class="n">urlparse</span>
+<span class="kn">import</span> <span class="nn">warnings</span>
+
+<span class="kn">import</span> <span class="nn">boto</span>
+<span class="kn">from</span> <span class="nn">boto.s3.connection</span> <span class="kn">import</span> <span class="n">S3Connection</span>
+<span class="kn">from</span> <span class="nn">boto.sts</span> <span class="kn">import</span> <span class="n">STSConnection</span>
+<span class="n">boto</span><span class="o">.</span><span class="n">set_stream_logger</span><span class="p">(</span><span class="s1">'boto'</span><span class="p">)</span>
+<span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s2">"boto"</span><span class="p">)</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">)</span>
+
+<span class="kn">from</span> <span class="nn">airflow.exceptions</span> <span class="kn">import</span> <span class="n">AirflowException</span>
+<span class="kn">from</span> <span class="nn">airflow.hooks.base_hook</span> <span class="kn">import</span> <span class="n">BaseHook</span>
+
+
+<span class="k">def</span> <span class="nf">_parse_s3_config</span><span class="p">(</span><span class="n">config_file_name</span><span class="p">,</span> <span class="n">config_format</span><span class="o">=</span><span class="s1">'boto'</span><span class="p">,</span> <span class="n">profile</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Parses a config file for s3 credentials. Can currently</span>
+<span class="sd"> parse boto, s3cmd.conf and AWS SDK config formats</span>
+
+<span class="sd"> :param config_file_name: path to the config file</span>
+<span class="sd"> :type config_file_name: str</span>
+<span class="sd"> :param config_format: config type. One of "boto", "s3cmd" or "aws".</span>
+<span class="sd"> Defaults to "boto"</span>
+<span class="sd"> :type config_format: str</span>
+<span class="sd"> :param profile: profile name in AWS type config file</span>
+<span class="sd"> :type profile: str</span>
+<span class="sd"> """</span>
+ <span class="n">Config</span> <span class="o">=</span> <span class="n">configparser</span><span class="o">.</span><span class="n">ConfigParser</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">Config</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">config_file_name</span><span class="p">):</span> <span class="c1"># pragma: no cover</span>
+ <span class="n">sections</span> <span class="o">=</span> <span class="n">Config</span><span class="o">.</span><span class="n">sections</span><span class="p">()</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"Couldn't read {0}"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">config_file_name</span><span class="p">))</span>
+ <span class="c1"># Setting option names depending on file format</span>
+ <span class="k">if</span> <span class="n">config_format</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
+ <span class="n">config_format</span> <span class="o">=</span> <span class="s1">'boto'</span>
+ <span class="n">conf_format</span> <span class="o">=</span> <span class="n">config_format</span><span class="o">.</span><span class="n">lower</span><span class="p">()</span>
+ <span class="k">if</span> <span class="n">conf_format</span> <span class="o">==</span> <span class="s1">'boto'</span><span class="p">:</span> <span class="c1"># pragma: no cover</span>
+ <span class="k">if</span> <span class="n">profile</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span> <span class="ow">and</span> <span class="s1">'profile '</span> <span class="o">+</span> <span class="n">profile</span> <span class="ow">in</span> <span class="n">sections</span><span class="p">:</span>
+ <span class="n">cred_section</span> <span class="o">=</span> <span class="s1">'profile '</span> <span class="o">+</span> <span class="n">profile</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">cred_section</span> <span class="o">=</span> <span class="s1">'Credentials'</span>
+ <span class="k">elif</span> <span class="n">conf_format</span> <span class="o">==</span> <span class="s1">'aws'</span> <span class="ow">and</span> <span class="n">profile</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
+ <span class="n">cred_section</span> <span class="o">=</span> <span class="n">profile</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">cred_section</span> <span class="o">=</span> <span class="s1">'default'</span>
+ <span class="c1"># Option names</span>
+ <span class="k">if</span> <span class="n">conf_format</span> <span class="ow">in</span> <span class="p">(</span><span class="s1">'boto'</span><span class="p">,</span> <span class="s1">'aws'</span><span class="p">):</span> <span class="c1"># pragma: no cover</span>
+ <span class="n">key_id_option</span> <span class="o">=</span> <span class="s1">'aws_access_key_id'</span>
+ <span class="n">secret_key_option</span> <span class="o">=</span> <span class="s1">'aws_secret_access_key'</span>
+ <span class="c1"># security_token_option = 'aws_security_token'</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">key_id_option</span> <span class="o">=</span> <span class="s1">'access_key'</span>
+ <span class="n">secret_key_option</span> <span class="o">=</span> <span class="s1">'secret_key'</span>
+ <span class="c1"># Actual Parsing</span>
+ <span class="k">if</span> <span class="n">cred_section</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">sections</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s2">"This config file format is not recognized"</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="n">access_key</span> <span class="o">=</span> <span class="n">Config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">cred_section</span><span class="p">,</span> <span class="n">key_id_option</span><span class="p">)</span>
+ <span class="n">secret_key</span> <span class="o">=</span> <span class="n">Config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">cred_section</span><span class="p">,</span> <span class="n">secret_key_option</span><span class="p">)</span>
+ <span class="n">calling_format</span> <span class="o">=</span> <span class="bp">None</span>
+ <span class="k">if</span> <span class="n">Config</span><span class="o">.</span><span class="n">has_option</span><span class="p">(</span><span class="n">cred_section</span><span class="p">,</span> <span class="s1">'calling_format'</span><span class="p">):</span>
+ <span class="n">calling_format</span> <span class="o">=</span> <span class="n">Config</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">cred_section</span><span class="p">,</span> <span class="s1">'calling_format'</span><span class="p">)</span>
+ <span class="k">except</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">warning</span><span class="p">(</span><span class="s2">"Option Error in parsing s3 config file"</span><span class="p">)</span>
+ <span class="k">raise</span>
+ <span class="k">return</span> <span class="p">(</span><span class="n">access_key</span><span class="p">,</span> <span class="n">secret_key</span><span class="p">,</span> <span class="n">calling_format</span><span class="p">)</span>
+
+
+<div class="viewcode-block" id="S3Hook"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook">[docs]</a><span class="k">class</span> <span class="nc">S3Hook</span><span class="p">(</span><span class="n">BaseHook</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Interact with S3. This class is a wrapper around the boto library.</span>
+<span class="sd"> """</span>
+ <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">s3_conn_id</span><span class="o">=</span><span class="s1">'s3_default'</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span> <span class="o">=</span> <span class="n">s3_conn_id</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_connection</span><span class="p">(</span><span class="n">s3_conn_id</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn</span><span class="o">.</span><span class="n">extra_dejson</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">profile</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'profile'</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">calling_format</span> <span class="o">=</span> <span class="bp">None</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_creds_in_conn</span> <span class="o">=</span> <span class="s1">'aws_secret_access_key'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_creds_in_config_file</span> <span class="o">=</span> <span class="s1">'s3_config_file'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_default_to_boto</span> <span class="o">=</span> <span class="bp">False</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_creds_in_conn</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_a_key</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">[</span><span class="s1">'aws_access_key_id'</span><span class="p">]</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_s_key</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">[</span><span class="s1">'aws_secret_access_key'</span><span class="p">]</span>
+ <span class="k">if</span> <span class="s1">'calling_format'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">calling_format</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">[</span><span class="s1">'calling_format'</span><span class="p">]</span>
+ <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">_creds_in_config_file</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3_config_file</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">[</span><span class="s1">'s3_config_file'</span><span class="p">]</span>
+ <span class="c1"># The format can be None and will default to boto in the parser</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3_config_format</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'s3_config_format'</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_default_to_boto</span> <span class="o">=</span> <span class="bp">True</span>
+ <span class="c1"># STS support for cross account resource access</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">_sts_conn_required</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'aws_account_id'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span> <span class="ow">or</span>
+ <span class="s1">'role_arn'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">)</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_sts_conn_required</span><span class="p">:</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">role_arn</span> <span class="o">=</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'role_arn'</span><span class="p">)</span> <span class="ow">or</span>
+ <span class="s2">"arn:aws:iam::"</span> <span class="o">+</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">[</span><span class="s1">'aws_account_id'</span><span class="p">]</span> <span class="o">+</span>
+ <span class="s2">":role/"</span> <span class="o">+</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">extra_params</span><span class="p">[</span><span class="s1">'aws_iam_role'</span><span class="p">])</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">connection</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+
+ <span class="k">def</span> <span class="nf">__getstate__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="n">pickled_dict</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">__dict__</span><span class="p">)</span>
+ <span class="k">del</span> <span class="n">pickled_dict</span><span class="p">[</span><span class="s1">'connection'</span><span class="p">]</span>
+ <span class="k">return</span> <span class="n">pickled_dict</span>
+
+ <span class="k">def</span> <span class="nf">__setstate__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">d</span><span class="p">):</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">__dict__</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">d</span><span class="p">)</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">__dict__</span><span class="p">[</span><span class="s1">'connection'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_conn</span><span class="p">()</span>
+
+ <span class="k">def</span> <span class="nf">_parse_s3_url</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">s3url</span><span class="p">):</span>
+ <span class="n">warnings</span><span class="o">.</span><span class="n">warn</span><span class="p">(</span>
+ <span class="s1">'Please note: S3Hook._parse_s3_url() is now '</span>
+ <span class="s1">'S3Hook.parse_s3_url() (no leading underscore).'</span><span class="p">,</span>
+ <span class="ne">DeprecationWarning</span><span class="p">)</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">s3url</span><span class="p">)</span>
+
+ <span class="nd">@staticmethod</span>
+ <span class="k">def</span> <span class="nf">parse_s3_url</span><span class="p">(</span><span class="n">s3url</span><span class="p">):</span>
+ <span class="n">parsed_url</span> <span class="o">=</span> <span class="n">urlparse</span><span class="p">(</span><span class="n">s3url</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">netloc</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="n">AirflowException</span><span class="p">(</span><span class="s1">'Please provide a bucket_name'</span><span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">bucket_name</span> <span class="o">=</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">netloc</span>
+ <span class="n">key</span> <span class="o">=</span> <span class="n">parsed_url</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">strip</span><span class="p">(</span><span class="s1">'/'</span><span class="p">)</span>
+ <span class="k">return</span> <span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span>
+
+<div class="viewcode-block" id="S3Hook.get_conn"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.get_conn">[docs]</a> <span class="k">def</span> <span class="nf">get_conn</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns the boto S3Connection object.</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_default_to_boto</span><span class="p">:</span>
+ <span class="k">return</span> <span class="n">S3Connection</span><span class="p">(</span><span class="n">profile_name</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">profile</span><span class="p">)</span>
+ <span class="n">a_key</span> <span class="o">=</span> <span class="n">s_key</span> <span class="o">=</span> <span class="bp">None</span>
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_creds_in_config_file</span><span class="p">:</span>
+ <span class="n">a_key</span><span class="p">,</span> <span class="n">s_key</span><span class="p">,</span> <span class="n">calling_format</span> <span class="o">=</span> <span class="n">_parse_s3_config</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">s3_config_file</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">s3_config_format</span><span class="p">,</span>
+ <span class="bp">self</span><span class="o">.</span><span class="n">profile</span><span class="p">)</span>
+ <span class="k">elif</span> <span class="bp">self</span><span class="o">.</span><span class="n">_creds_in_conn</span><span class="p">:</span>
+ <span class="n">a_key</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_a_key</span>
+ <span class="n">s_key</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_s_key</span>
+ <span class="n">calling_format</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">calling_format</span>
+
+ <span class="k">if</span> <span class="n">calling_format</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
+ <span class="n">calling_format</span> <span class="o">=</span> <span class="s1">'boto.s3.connection.SubdomainCallingFormat'</span>
+
+ <span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_sts_conn_required</span><span class="p">:</span>
+ <span class="n">sts_connection</span> <span class="o">=</span> <span class="n">STSConnection</span><span class="p">(</span><span class="n">aws_access_key_id</span><span class="o">=</span><span class="n">a_key</span><span class="p">,</span>
+ <span class="n">aws_secret_access_key</span><span class="o">=</span><span class="n">s_key</span><span class="p">,</span>
+ <span class="n">profile_name</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">profile</span><span class="p">)</span>
+ <span class="n">assumed_role_object</span> <span class="o">=</span> <span class="n">sts_connection</span><span class="o">.</span><span class="n">assume_role</span><span class="p">(</span>
+ <span class="n">role_arn</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">role_arn</span><span class="p">,</span>
+ <span class="n">role_session_name</span><span class="o">=</span><span class="s2">"Airflow_"</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">s3_conn_id</span>
+ <span class="p">)</span>
+ <span class="n">creds</span> <span class="o">=</span> <span class="n">assumed_role_object</span><span class="o">.</span><span class="n">credentials</span>
+ <span class="n">connection</span> <span class="o">=</span> <span class="n">S3Connection</span><span class="p">(</span>
+ <span class="n">aws_access_key_id</span><span class="o">=</span><span class="n">creds</span><span class="o">.</span><span class="n">access_key</span><span class="p">,</span>
+ <span class="n">aws_secret_access_key</span><span class="o">=</span><span class="n">creds</span><span class="o">.</span><span class="n">secret_key</span><span class="p">,</span>
+ <span class="n">calling_format</span><span class="o">=</span><span class="n">calling_format</span><span class="p">,</span>
+ <span class="n">security_token</span><span class="o">=</span><span class="n">creds</span><span class="o">.</span><span class="n">session_token</span>
+ <span class="p">)</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="n">connection</span> <span class="o">=</span> <span class="n">S3Connection</span><span class="p">(</span><span class="n">aws_access_key_id</span><span class="o">=</span><span class="n">a_key</span><span class="p">,</span>
+ <span class="n">aws_secret_access_key</span><span class="o">=</span><span class="n">s_key</span><span class="p">,</span>
+ <span class="n">calling_format</span><span class="o">=</span><span class="n">calling_format</span><span class="p">,</span>
+ <span class="n">profile_name</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">profile</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">connection</span></div>
+
+<div class="viewcode-block" id="S3Hook.check_for_bucket"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.check_for_bucket">[docs]</a> <span class="k">def</span> <span class="nf">check_for_bucket</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Check if bucket_name exists.</span>
+
+<span class="sd"> :param bucket_name: the name of the bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> """</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">lookup</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span></div>
+
+<div class="viewcode-block" id="S3Hook.get_bucket"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.get_bucket">[docs]</a> <span class="k">def</span> <span class="nf">get_bucket</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a boto.s3.bucket.Bucket object</span>
+
+<span class="sd"> :param bucket_name: the name of the bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> """</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">connection</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="S3Hook.list_keys"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.list_keys">[docs]</a> <span class="k">def</span> <span class="nf">list_keys</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Lists keys in a bucket under prefix and not containing delimiter</span>
+
+<span class="sd"> :param bucket_name: the name of the bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> :param prefix: a key prefix</span>
+<span class="sd"> :type prefix: str</span>
+<span class="sd"> :param delimiter: the delimiter marks key hierarchy.</span>
+<span class="sd"> :type delimiter: str</span>
+<span class="sd"> """</span>
+ <span class="n">b</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span>
+ <span class="n">keylist</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">b</span><span class="o">.</span><span class="n">list</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">))</span>
+ <span class="k">return</span> <span class="p">[</span><span class="n">k</span><span class="o">.</span><span class="n">name</span> <span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">keylist</span><span class="p">]</span> <span class="k">if</span> <span class="n">keylist</span> <span class="o">!=</span> <span class="p">[]</span> <span class="k">else</span> <span class="bp">None</span></div>
+
+<div class="viewcode-block" id="S3Hook.list_prefixes"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.list_prefixes">[docs]</a> <span class="k">def</span> <span class="nf">list_prefixes</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Lists prefixes in a bucket under prefix</span>
+
+<span class="sd"> :param bucket_name: the name of the bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> :param prefix: a key prefix</span>
+<span class="sd"> :type prefix: str</span>
+<span class="sd"> :param delimiter: the delimiter marks key hierarchy.</span>
+<span class="sd"> :type delimiter: str</span>
+<span class="sd"> """</span>
+ <span class="n">b</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span>
+ <span class="n">plist</span> <span class="o">=</span> <span class="n">b</span><span class="o">.</span><span class="n">list</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">)</span>
+ <span class="n">prefix_names</span> <span class="o">=</span> <span class="p">[</span><span class="n">p</span><span class="o">.</span><span class="n">name</span> <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">plist</span>
+ <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">boto</span><span class="o">.</span><span class="n">s3</span><span class="o">.</span><span class="n">prefix</span><span class="o">.</span><span class="n">Prefix</span><span class="p">)]</span>
+ <span class="k">return</span> <span class="n">prefix_names</span> <span class="k">if</span> <span class="n">prefix_names</span> <span class="o">!=</span> <span class="p">[]</span> <span class="k">else</span> <span class="bp">None</span></div>
+
+<div class="viewcode-block" id="S3Hook.check_for_key"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.check_for_key">[docs]</a> <span class="k">def</span> <span class="nf">check_for_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Checks that a key exists in a bucket</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">bucket_name</span><span class="p">:</span>
+ <span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
+ <span class="n">bucket</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">bucket</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span></div>
+
+<div class="viewcode-block" id="S3Hook.get_key"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.get_key">[docs]</a> <span class="k">def</span> <span class="nf">get_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a boto.s3.key.Key object</span>
+
+<span class="sd"> :param key: the path to the key</span>
+<span class="sd"> :type key: str</span>
+<span class="sd"> :param bucket_name: the name of the bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">bucket_name</span><span class="p">:</span>
+ <span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
+ <span class="n">bucket</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span>
+ <span class="k">return</span> <span class="n">bucket</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="n">key</span><span class="p">)</span></div>
+
+<div class="viewcode-block" id="S3Hook.check_for_wildcard_key"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.check_for_wildcard_key">[docs]</a> <span class="k">def</span> <span class="nf">check_for_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span>
+ <span class="n">wildcard_key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Checks that a key matching a wildcard expression exists in a bucket</span>
+<span class="sd"> """</span>
+ <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_wildcard_key</span><span class="p">(</span><span class="n">wildcard_key</span><span class="o">=</span><span class="n">wildcard_key</span><span class="p">,</span>
+ <span class="n">bucket_name</span><span class="o">=</span><span class="n">bucket_name</span><span class="p">,</span>
+ <span class="n">delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">)</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span></div>
+
+<div class="viewcode-block" id="S3Hook.get_wildcard_key"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.get_wildcard_key">[docs]</a> <span class="k">def</span> <span class="nf">get_wildcard_key</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">wildcard_key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="s1">''</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Returns a boto.s3.key.Key object matching the regular expression</span>
+
+<span class="sd"> :param regex_key: the path to the key</span>
+<span class="sd"> :type regex_key: str</span>
+<span class="sd"> :param bucket_name: the name of the bucket</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">bucket_name</span><span class="p">:</span>
+ <span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">wildcard_key</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">wildcard_key</span><span class="p">)</span>
+ <span class="n">bucket</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span>
+ <span class="n">prefix</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">r'[*]'</span><span class="p">,</span> <span class="n">wildcard_key</span><span class="p">,</span> <span class="mi">1</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
+ <span class="n">klist</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">list_keys</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="o">=</span><span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="o">=</span><span class="n">delimiter</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">klist</span><span class="p">:</span>
+ <span class="k">return</span> <span class="bp">None</span>
+ <span class="n">key_matches</span> <span class="o">=</span> <span class="p">[</span><span class="n">k</span> <span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">klist</span> <span class="k">if</span> <span class="n">fnmatch</span><span class="o">.</span><span class="n">fnmatch</span><span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">wildcard_key</span><span class="p">)]</span>
+ <span class="k">return</span> <span class="n">bucket</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="n">key_matches</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="k">if</span> <span class="n">key_matches</span> <span class="k">else</span> <span class="bp">None</span></div>
+
+<div class="viewcode-block" id="S3Hook.check_for_prefix"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.check_for_prefix">[docs]</a> <span class="k">def</span> <span class="nf">check_for_prefix</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">bucket_name</span><span class="p">,</span> <span class="n">prefix</span><span class="p">,</span> <span class="n">delimiter</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Checks that a prefix exists in a bucket</span>
+<span class="sd"> """</span>
+ <span class="n">prefix</span> <span class="o">=</span> <span class="n">prefix</span> <span class="o">+</span> <span class="n">delimiter</span> <span class="k">if</span> <span class="n">prefix</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">!=</span> <span class="n">delimiter</span> <span class="k">else</span> <span class="n">prefix</span>
+ <span class="n">prefix_split</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">r'(\w+[{d}])$'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">d</span><span class="o">=</span><span class="n">delimiter</span><span class="p">),</span> <span class="n">prefix</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
+ <span class="n">previous_level</span> <span class="o">=</span> <span class="n">prefix_split</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
+ <span class="n">plist</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">list_prefixes</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">previous_level</span><span class="p">,</span> <span class="n">delimiter</span><span class="p">)</span>
+ <span class="k">return</span> <span class="bp">False</span> <span class="k">if</span> <span class="n">plist</span> <span class="ow">is</span> <span class="bp">None</span> <span class="k">else</span> <span class="n">prefix</span> <span class="ow">in</span> <span class="n">plist</span></div>
+
+<div class="viewcode-block" id="S3Hook.load_file"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.load_file">[docs]</a> <span class="k">def</span> <span class="nf">load_file</span><span class="p">(</span>
+ <span class="bp">self</span><span class="p">,</span>
+ <span class="n">filename</span><span class="p">,</span>
+ <span class="n">key</span><span class="p">,</span>
+ <span class="n">bucket_name</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">replace</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">multipart_bytes</span><span class="o">=</span><span class="mi">5</span> <span class="o">*</span> <span class="p">(</span><span class="mi">1024</span> <span class="o">**</span> <span class="mi">3</span><span class="p">)):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Loads a local file to S3</span>
+
+<span class="sd"> :param filename: name of the file to load.</span>
+<span class="sd"> :type filename: str</span>
+<span class="sd"> :param key: S3 key that will point to the file</span>
+<span class="sd"> :type key: str</span>
+<span class="sd"> :param bucket_name: Name of the bucket in which to store the file</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> :param replace: A flag to decide whether or not to overwrite the key</span>
+<span class="sd"> if it already exists. If replace is False and the key exists, an</span>
+<span class="sd"> error will be raised.</span>
+<span class="sd"> :type replace: bool</span>
+<span class="sd"> :param multipart_bytes: If provided, the file is uploaded in parts of</span>
+<span class="sd"> this size (minimum 5242880). The default value is 5GB, since S3</span>
+<span class="sd"> cannot accept non-multipart uploads for files larger than 5GB. If</span>
+<span class="sd"> the file is smaller than the specified limit, the option will be</span>
+<span class="sd"> ignored.</span>
+<span class="sd"> :type multipart_bytes: int</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">bucket_name</span><span class="p">:</span>
+ <span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
+ <span class="n">bucket</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span>
+ <span class="n">key_obj</span> <span class="o">=</span> <span class="n">bucket</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">replace</span> <span class="ow">and</span> <span class="n">key_obj</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"The key {key} already exists."</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+
+ <span class="n">key_size</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">getsize</span><span class="p">(</span><span class="n">filename</span><span class="p">)</span>
+ <span class="k">if</span> <span class="n">multipart_bytes</span> <span class="ow">and</span> <span class="n">key_size</span> <span class="o">>=</span> <span class="n">multipart_bytes</span><span class="p">:</span>
+ <span class="c1"># multipart upload</span>
+ <span class="kn">from</span> <span class="nn">filechunkio</span> <span class="kn">import</span> <span class="n">FileChunkIO</span>
+ <span class="n">mp</span> <span class="o">=</span> <span class="n">bucket</span><span class="o">.</span><span class="n">initiate_multipart_upload</span><span class="p">(</span><span class="n">key_name</span><span class="o">=</span><span class="n">key</span><span class="p">)</span>
+ <span class="n">total_chunks</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">math</span><span class="o">.</span><span class="n">ceil</span><span class="p">(</span><span class="n">key_size</span> <span class="o">/</span> <span class="n">multipart_bytes</span><span class="p">))</span>
+ <span class="n">sent_bytes</span> <span class="o">=</span> <span class="mi">0</span>
+ <span class="k">try</span><span class="p">:</span>
+ <span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">total_chunks</span><span class="p">):</span>
+ <span class="n">offset</span> <span class="o">=</span> <span class="n">chunk</span> <span class="o">*</span> <span class="n">multipart_bytes</span>
+ <span class="nb">bytes</span> <span class="o">=</span> <span class="nb">min</span><span class="p">(</span><span class="n">multipart_bytes</span><span class="p">,</span> <span class="n">key_size</span> <span class="o">-</span> <span class="n">offset</span><span class="p">)</span>
+ <span class="k">with</span> <span class="n">FileChunkIO</span><span class="p">(</span>
+ <span class="n">filename</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">,</span> <span class="n">offset</span><span class="o">=</span><span class="n">offset</span><span class="p">,</span> <span class="nb">bytes</span><span class="o">=</span><span class="nb">bytes</span><span class="p">)</span> <span class="k">as</span> <span class="n">fp</span><span class="p">:</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s1">'Sending chunk {c} of {tc}...'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="n">c</span><span class="o">=</span><span class="n">chunk</span> <span class="o">+</span> <span class="mi">1</span><span class="p">,</span> <span class="n">tc</span><span class="o">=</span><span class="n">total_chunks</span><span class="p">))</span>
+ <span class="n">mp</span><span class="o">.</span><span class="n">upload_part_from_file</span><span class="p">(</span><span class="n">fp</span><span class="p">,</span> <span class="n">part_num</span><span class="o">=</span><span class="n">chunk</span> <span class="o">+</span> <span class="mi">1</span><span class="p">)</span>
+ <span class="k">except</span><span class="p">:</span>
+ <span class="n">mp</span><span class="o">.</span><span class="n">cancel_upload</span><span class="p">()</span>
+ <span class="k">raise</span>
+ <span class="n">mp</span><span class="o">.</span><span class="n">complete_upload</span><span class="p">()</span>
+ <span class="k">else</span><span class="p">:</span>
+ <span class="c1"># regular upload</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">key_obj</span><span class="p">:</span>
+ <span class="n">key_obj</span> <span class="o">=</span> <span class="n">bucket</span><span class="o">.</span><span class="n">new_key</span><span class="p">(</span><span class="n">key_name</span><span class="o">=</span><span class="n">key</span><span class="p">)</span>
+ <span class="n">key_size</span> <span class="o">=</span> <span class="n">key_obj</span><span class="o">.</span><span class="n">set_contents_from_filename</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span>
+ <span class="n">replace</span><span class="o">=</span><span class="n">replace</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"The key {key} now contains"</span>
+ <span class="s2">" {key_size} bytes"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span></div>
+
+<div class="viewcode-block" id="S3Hook.load_string"><a class="viewcode-back" href="../code.html#airflow.hooks.S3Hook.load_string">[docs]</a> <span class="k">def</span> <span class="nf">load_string</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">string_data</span><span class="p">,</span>
+ <span class="n">key</span><span class="p">,</span> <span class="n">bucket_name</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
+ <span class="n">replace</span><span class="o">=</span><span class="bp">False</span><span class="p">,</span>
+ <span class="n">encrypt</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
+ <span class="sd">"""</span>
+<span class="sd"> Loads a local file to S3</span>
+
+<span class="sd"> This is provided as a convenience to drop a file in S3. It uses the</span>
+<span class="sd"> boto infrastructure to ship a file to s3. It is currently using only</span>
+<span class="sd"> a single part download, and should not be used to move large files.</span>
+
+<span class="sd"> :param string_data: string to set as content for the key.</span>
+<span class="sd"> :type string_data: str</span>
+<span class="sd"> :param key: S3 key that will point to the file</span>
+<span class="sd"> :type key: str</span>
+<span class="sd"> :param bucket_name: Name of the bucket in which to store the file</span>
+<span class="sd"> :type bucket_name: str</span>
+<span class="sd"> :param replace: A flag to decide whether or not to overwrite the key</span>
+<span class="sd"> if it already exists</span>
+<span class="sd"> :type replace: bool</span>
+<span class="sd"> """</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">bucket_name</span><span class="p">:</span>
+ <span class="p">(</span><span class="n">bucket_name</span><span class="p">,</span> <span class="n">key</span><span class="p">)</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parse_s3_url</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
+ <span class="n">bucket</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get_bucket</span><span class="p">(</span><span class="n">bucket_name</span><span class="p">)</span>
+ <span class="n">key_obj</span> <span class="o">=</span> <span class="n">bucket</span><span class="o">.</span><span class="n">get_key</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">replace</span> <span class="ow">and</span> <span class="n">key_obj</span><span class="p">:</span>
+ <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s2">"The key {key} already exists."</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
+ <span class="o">**</span><span class="nb">locals</span><span class="p">()))</span>
+ <span class="k">if</span> <span class="ow">not</span> <span class="n">key_obj</span><span class="p">:</span>
+ <span class="n">key_obj</span> <span class="o">=</span> <span class="n">bucket</span><span class="o">.</span><span class="n">new_key</span><span class="p">(</span><span class="n">key_name</span><span class="o">=</span><span class="n">key</span><span class="p">)</span>
+ <span class="n">key_size</span> <span class="o">=</span> <span class="n">key_obj</span><span class="o">.</span><span class="n">set_contents_from_string</span><span class="p">(</span><span class="n">string_data</span><span class="p">,</span>
+ <span class="n">replace</span><span class="o">=</span><span class="n">replace</span><span class="p">,</span>
+ <span class="n">encrypt_key</span><span class="o">=</span><span class="n">encrypt</span><span class="p">)</span>
+ <span class="n">logging</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s2">"The key {key} now contains"</span>
+ <span class="s2">" {key_size} bytes"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="nb">locals</span><span class="p">()))</span></div></div>
+</pre></div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'../',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="../_static/jquery.js"></script>
+ <script type="text/javascript" src="../_static/underscore.js"></script>
+ <script type="text/javascript" src="../_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="../_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file