You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ma...@apache.org on 2016/06/05 05:23:52 UTC
[02/34] incubator-airflow-site git commit: Initial commit
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/scheduler.html
----------------------------------------------------------------------
diff --git a/scheduler.html b/scheduler.html
new file mode 100644
index 0000000..4c703f8
--- /dev/null
+++ b/scheduler.html
@@ -0,0 +1,328 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Scheduling & Triggers — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="Plugins" href="plugins.html"/>
+ <link rel="prev" title="Command Line Interface" href="cli.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">Scheduling & Triggers</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#dag-runs">DAG Runs</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#external-triggers">External Triggers</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#to-keep-in-mind">To Keep in Mind</a></li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>Scheduling & Triggers</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/scheduler.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="scheduling-triggers">
+<h1>Scheduling & Triggers<a class="headerlink" href="#scheduling-triggers" title="Permalink to this headline">�</a></h1>
+<p>The Airflow scheduler monitors all tasks and all DAGs, and triggers the
+task instances whose dependencies have been met. Behind the scenes,
+it monitors and stays in sync with a folder for all DAG objects it may contain,
+and periodically (every minute or so) inspects active tasks to see whether
+they can be triggered.</p>
+<p>The Airflow scheduler is designed to run as a persistent service in an
+Airflow production environment. To kick it off, all you need to do is
+execute <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">scheduler</span></code>. It will use the configuration specified in
+<code class="docutils literal"><span class="pre">airflow.cfg</span></code>.</p>
+<p>Note that if you run a DAG on a <code class="docutils literal"><span class="pre">schedule_interval</span></code> of one day,
+the run stamped <code class="docutils literal"><span class="pre">2016-01-01</span></code> will be trigger soon after <code class="docutils literal"><span class="pre">2016-01-01T23:59</span></code>.
+In other words, the job instance is started once the period it covers
+has ended.</p>
+<p>The scheduler starts an instance of the executor specified in the your
+<code class="docutils literal"><span class="pre">airflow.cfg</span></code>. If it happens to be the <code class="docutils literal"><span class="pre">LocalExecutor</span></code>, tasks will be
+executed as subprocesses; in the case of <code class="docutils literal"><span class="pre">CeleryExecutor</span></code> and
+<code class="docutils literal"><span class="pre">MesosExecutor</span></code>, tasks are executed remotely.</p>
+<p>To start a scheduler, simply run the command:</p>
+<div class="code bash highlight-default"><div class="highlight"><pre><span></span><span class="n">airflow</span> <span class="n">scheduler</span>
+</pre></div>
+</div>
+<div class="section" id="dag-runs">
+<h2>DAG Runs<a class="headerlink" href="#dag-runs" title="Permalink to this headline">�</a></h2>
+<p>A DAG Run is an object representing an instantiation of the DAG in time.</p>
+<p>Each DAG may or may not have a schedule, which informs how <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Runs</span></code> are
+created. <code class="docutils literal"><span class="pre">schedule_interval</span></code> is defined as a DAG arguments, and receives
+preferably a
+<a class="reference external" href="https://en.wikipedia.org/wiki/Cron#CRON_expression">cron expression</a> as
+a <code class="docutils literal"><span class="pre">str</span></code>, or a <code class="docutils literal"><span class="pre">datetime.timedelta</span></code> object. Alternatively, you can also
+use one of these cron “preset”:</p>
+<table border="1" class="docutils">
+<colgroup>
+<col width="15%" />
+<col width="69%" />
+<col width="16%" />
+</colgroup>
+<thead valign="bottom">
+<tr class="row-odd"><th class="head">preset</th>
+<th class="head">Run once a year at midnight of January 1</th>
+<th class="head">cron</th>
+</tr>
+</thead>
+<tbody valign="top">
+<tr class="row-even"><td><code class="docutils literal"><span class="pre">None</span></code></td>
+<td>Don’t schedule, use for exclusively “externally triggered”
+DAGs</td>
+<td> </td>
+</tr>
+<tr class="row-odd"><td><code class="docutils literal"><span class="pre">@once</span></code></td>
+<td>Schedule once and only once</td>
+<td> </td>
+</tr>
+<tr class="row-even"><td><code class="docutils literal"><span class="pre">@hourly</span></code></td>
+<td>Run once an hour at the beginning of the hour</td>
+<td><code class="docutils literal"><span class="pre">0</span> <span class="pre">*</span> <span class="pre">*</span> <span class="pre">*</span> <span class="pre">*</span></code></td>
+</tr>
+<tr class="row-odd"><td><code class="docutils literal"><span class="pre">@daily</span></code></td>
+<td>Run once a day at midnight</td>
+<td><code class="docutils literal"><span class="pre">0</span> <span class="pre">0</span> <span class="pre">*</span> <span class="pre">*</span> <span class="pre">*</span></code></td>
+</tr>
+<tr class="row-even"><td><code class="docutils literal"><span class="pre">@weekly</span></code></td>
+<td>Run once a week at midnight on Sunday morning</td>
+<td><code class="docutils literal"><span class="pre">0</span> <span class="pre">0</span> <span class="pre">*</span> <span class="pre">*</span> <span class="pre">0</span></code></td>
+</tr>
+<tr class="row-odd"><td><code class="docutils literal"><span class="pre">@monthly</span></code></td>
+<td>Run once a month at midnight of the first day of the month</td>
+<td><code class="docutils literal"><span class="pre">0</span> <span class="pre">0</span> <span class="pre">1</span> <span class="pre">*</span> <span class="pre">*</span></code></td>
+</tr>
+<tr class="row-even"><td><code class="docutils literal"><span class="pre">@yearly</span></code></td>
+<td>Run once a year at midnight of January 1</td>
+<td><code class="docutils literal"><span class="pre">0</span> <span class="pre">0</span> <span class="pre">1</span> <span class="pre">1</span> <span class="pre">*</span></code></td>
+</tr>
+</tbody>
+</table>
+<p>Your DAG will be instantiated
+for each schedule, while creating a <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Run</span></code> entry for each schedule.</p>
+<p>DAG runs have a state associated to them (running, failed, success) and
+informs the scheduler on which set of schedules should be evaluated for
+task submissions. Without the metadata at the DAG run level, the Airflow
+scheduler would have much more work to do in order to figure out what tasks
+should be triggered and come to a crawl. It might also create undesired
+processing when changing the shape of your DAG, by say adding in new
+tasks.</p>
+</div>
+<div class="section" id="external-triggers">
+<h2>External Triggers<a class="headerlink" href="#external-triggers" title="Permalink to this headline">�</a></h2>
+<p>Note that <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Runs</span></code> can also be created manually through the CLI while
+running an <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">trigger_dag</span></code> command, where you can define a
+specific <code class="docutils literal"><span class="pre">run_id</span></code>. The <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Runs</span></code> created externally to the
+scheduler get associated to the trigger’s timestamp, and will be displayed
+in the UI alongside scheduled <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">runs</span></code>.</p>
+</div>
+<div class="section" id="to-keep-in-mind">
+<h2>To Keep in Mind<a class="headerlink" href="#to-keep-in-mind" title="Permalink to this headline">�</a></h2>
+<ul class="simple">
+<li>The first <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Run</span></code> is created based on the minimum <code class="docutils literal"><span class="pre">start_date</span></code> for the
+tasks in your DAG.</li>
+<li>Subsequent <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Runs</span></code> are created by the scheduler process, based on
+your DAG’s <code class="docutils literal"><span class="pre">schedule_interval</span></code>, sequentially.</li>
+<li>When clearing a set of tasks’ state in hope of getting them to re-run,
+it is important to keep in mind the <code class="docutils literal"><span class="pre">DAG</span> <span class="pre">Run</span></code>‘s state too as it defines
+whether the scheduler should look into triggering tasks for that run.</li>
+</ul>
+<p>Here are some of the ways you can <strong>unblock tasks</strong>:</p>
+<ul class="simple">
+<li>From the UI, you can <strong>clear</strong> (as in delete the status of) individual task instances from the task instances dialog, while defining whether you want to includes the past/future and the upstream/downstream dependencies. Note that a confirmation window comes next and allows you to see the set you are about to clear.</li>
+<li>The CLI command <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">clear</span> <span class="pre">-h</span></code> has lots of options when it comes to clearing task instance states, including specifying date ranges, targeting task_ids by specifying a regular expression, flags for including upstream and downstream relatives, and targeting task instances in specific states (<code class="docutils literal"><span class="pre">failed</span></code>, or <code class="docutils literal"><span class="pre">success</span></code>)</li>
+<li>Marking task instances as successful can be done through the UI. This is mostly to fix false negatives, or for instance when the fix has been applied outside of Airflow.</li>
+<li>The <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">backfill</span></code> CLI subcommand has a flag to <code class="docutils literal"><span class="pre">--mark_success</span></code> and allows selecting subsections of the DAG as well as specifying date ranges.</li>
+</ul>
+</div>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="plugins.html" class="btn btn-neutral float-right" title="Plugins" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="cli.html" class="btn btn-neutral" title="Command Line Interface" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/search.html
----------------------------------------------------------------------
diff --git a/search.html b/search.html
new file mode 100644
index 0000000..cdf8e86
--- /dev/null
+++ b/search.html
@@ -0,0 +1,214 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Search — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="#" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul>
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li></li>
+ <li class="wy-breadcrumbs-aside">
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <noscript>
+ <div id="fallback" class="admonition warning">
+ <p class="last">
+ Please activate JavaScript to enable the search
+ functionality.
+ </p>
+ </div>
+ </noscript>
+
+
+ <div id="search-results">
+
+ </div>
+
+ </div>
+ </div>
+ <footer>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+ <script type="text/javascript" src="_static/searchtools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+ <script type="text/javascript">
+ jQuery(function() { Search.loadIndex("searchindex.js"); });
+ </script>
+
+ <script type="text/javascript" id="searchindexloader"></script>
+
+
+
+</body>
+</html>
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/searchindex.js
----------------------------------------------------------------------
diff --git a/searchindex.js b/searchindex.js
new file mode 100644
index 0000000..434ea1e
--- /dev/null
+++ b/searchindex.js
@@ -0,0 +1 @@
+Search.setIndex({envversion:47,filenames:["cli","code","concepts","configuration","faq","index","installation","license","plugins","profiling","project","scheduler","security","start","tutorial","ui"],objects:{"airflow.contrib":{executors:[1,0,0,"-"],hooks:[1,0,0,"-"],operators:[1,0,0,"-"]},"airflow.contrib.hooks":{CloudantHook:[1,1,1,""],FTPHook:[1,1,1,""],SSHHook:[1,1,1,""],VerticaHook:[1,1,1,""]},"airflow.contrib.hooks.CloudantHook":{db:[1,2,1,""]},"airflow.contrib.hooks.FTPHook":{close_conn:[1,2,1,""],create_directory:[1,2,1,""],delete_directory:[1,2,1,""],delete_file:[1,2,1,""],describe_directory:[1,2,1,""],get_conn:[1,2,1,""],list_directory:[1,2,1,""],retrieve_file:[1,2,1,""],store_file:[1,2,1,""]},"airflow.contrib.hooks.SSHHook":{Popen:[1,2,1,""],check_output:[1,2,1,""],tunnel:[1,2,1,""]},"airflow.contrib.hooks.VerticaHook":{get_conn:[1,2,1,""]},"airflow.contrib.operators":{SSHExecuteOperator:[1,1,1,""],VerticaOperator:[1,1,1,""],VerticaToHiveTransfer:[1,1,1,""]},"airflow.con
trib.operators.hipchat_operator":{HipChatAPIOperator:[1,1,1,""],HipChatAPISendRoomNotificationOperator:[1,1,1,""]},"airflow.executors":{CeleryExecutor:[1,1,1,""],LocalExecutor:[1,1,1,""],SequentialExecutor:[1,1,1,""]},"airflow.hooks":{DbApiHook:[1,1,1,""],DruidHook:[1,1,1,""],HiveCliHook:[1,1,1,""],HiveMetastoreHook:[1,1,1,""],HiveServer2Hook:[1,1,1,""],HttpHook:[1,1,1,""],MsSqlHook:[1,1,1,""],MySqlHook:[1,1,1,""],PostgresHook:[1,1,1,""],PrestoHook:[1,1,1,""],S3Hook:[1,1,1,""],SqliteHook:[1,1,1,""],WebHDFSHook:[1,1,1,""]},"airflow.hooks.DbApiHook":{bulk_dump:[1,2,1,""],bulk_load:[1,2,1,""],get_conn:[1,2,1,""],get_cursor:[1,2,1,""],get_first:[1,2,1,""],get_pandas_df:[1,2,1,""],get_records:[1,2,1,""],insert_rows:[1,2,1,""],run:[1,2,1,""]},"airflow.hooks.DruidHook":{construct_ingest_query:[1,2,1,""],get_conn:[1,2,1,""],load_from_hdfs:[1,2,1,""]},"airflow.hooks.HiveCliHook":{load_file:[1,2,1,""],run_cli:[1,2,1,""],test_hql:[1,2,1,""]},"airflow.hooks.HiveMetastoreHook":{check_for_partiti
on:[1,2,1,""],get_databases:[1,2,1,""],get_metastore_client:[1,2,1,""],get_partitions:[1,2,1,""],get_table:[1,2,1,""],get_tables:[1,2,1,""],max_partition:[1,2,1,""],table_exists:[1,2,1,""]},"airflow.hooks.HiveServer2Hook":{get_pandas_df:[1,2,1,""],get_records:[1,2,1,""]},"airflow.hooks.HttpHook":{get_conn:[1,2,1,""],run:[1,2,1,""],run_and_check:[1,2,1,""]},"airflow.hooks.MsSqlHook":{get_conn:[1,2,1,""]},"airflow.hooks.MySqlHook":{bulk_load:[1,2,1,""],get_conn:[1,2,1,""]},"airflow.hooks.PrestoHook":{get_conn:[1,2,1,""],get_first:[1,2,1,""],get_pandas_df:[1,2,1,""],get_records:[1,2,1,""],run:[1,2,1,""]},"airflow.hooks.S3Hook":{check_for_bucket:[1,2,1,""],check_for_key:[1,2,1,""],check_for_prefix:[1,2,1,""],check_for_wildcard_key:[1,2,1,""],get_bucket:[1,2,1,""],get_conn:[1,2,1,""],get_key:[1,2,1,""],get_wildcard_key:[1,2,1,""],list_keys:[1,2,1,""],list_prefixes:[1,2,1,""],load_file:[1,2,1,""],load_string:[1,2,1,""]},"airflow.hooks.SqliteHook":{get_conn:[1,2,1,""]},"airflow.hooks.WebHD
FSHook":{check_for_path:[1,2,1,""],get_conn:[1,2,1,""],load_file:[1,2,1,""]},"airflow.macros":{ds_add:[1,3,1,""],ds_format:[1,3,1,""],hive:[1,0,0,"-"],integrate_plugins:[1,3,1,""],random:[1,3,1,""]},"airflow.macros.hive":{closest_ds_partition:[1,3,1,""],max_partition:[1,3,1,""]},"airflow.models":{BaseOperator:[1,1,1,""],Connection:[1,1,1,""],DAG:[1,1,1,""],DagBag:[1,1,1,""],TaskInstance:[1,1,1,""]},"airflow.models.BaseOperator":{clear:[1,2,1,""],dag:[1,4,1,""],detect_downstream_cycle:[1,2,1,""],downstream_list:[1,4,1,""],execute:[1,2,1,""],get_direct_relatives:[1,2,1,""],get_flat_relatives:[1,2,1,""],get_task_instances:[1,2,1,""],has_dag:[1,2,1,""],on_kill:[1,2,1,""],post_execute:[1,2,1,""],pre_execute:[1,2,1,""],prepare_template:[1,2,1,""],render_template:[1,2,1,""],render_template_from_field:[1,2,1,""],run:[1,2,1,""],schedule_interval:[1,4,1,""],set_downstream:[1,2,1,""],set_upstream:[1,2,1,""],upstream_list:[1,4,1,""],xcom_pull:[1,2,1,""],xcom_push:[1,2,1,""]},"airflow.models.Con
nection":{extra_dejson:[1,4,1,""]},"airflow.models.DAG":{add_task:[1,2,1,""],add_tasks:[1,2,1,""],cli:[1,2,1,""],concurrency_reached:[1,4,1,""],crawl_for_tasks:[1,2,1,""],create_dagrun:[1,2,1,""],filepath:[1,4,1,""],folder:[1,4,1,""],get_template_env:[1,2,1,""],is_paused:[1,4,1,""],latest_execution_date:[1,4,1,""],run:[1,2,1,""],set_dependency:[1,2,1,""],sub_dag:[1,2,1,""],subdags:[1,4,1,""],tree_view:[1,2,1,""]},"airflow.models.DagBag":{bag_dag:[1,2,1,""],collect_dags:[1,2,1,""],dagbag_report:[1,2,1,""],get_dag:[1,2,1,""],kill_zombies:[1,2,1,""],process_file:[1,2,1,""],size:[1,2,1,""]},"airflow.models.TaskInstance":{are_dependencies_met:[1,2,1,""],are_dependents_done:[1,2,1,""],clear_xcom_data:[1,2,1,""],command:[1,2,1,""],current_state:[1,2,1,""],error:[1,2,1,""],evaluate_trigger_rule:[1,2,1,""],is_premature:[1,2,1,""],is_queueable:[1,2,1,""],is_runnable:[1,2,1,""],key:[1,4,1,""],pool_full:[1,2,1,""],ready_for_retry:[1,2,1,""],refresh_from_db:[1,2,1,""],run:[1,2,1,""],xcom_pull:[1
,2,1,""],xcom_push:[1,2,1,""]},"airflow.operators":{BashOperator:[1,1,1,""],BranchPythonOperator:[1,1,1,""],DummyOperator:[1,1,1,""],EmailOperator:[1,1,1,""],ExternalTaskSensor:[1,1,1,""],GenericTransfer:[1,1,1,""],HdfsSensor:[1,1,1,""],Hive2SambaOperator:[1,1,1,""],HiveOperator:[1,1,1,""],HivePartitionSensor:[1,1,1,""],HiveToDruidTransfer:[1,1,1,""],HiveToMySqlTransfer:[1,1,1,""],HttpSensor:[1,1,1,""],MetastorePartitionSensor:[1,1,1,""],MsSqlOperator:[1,1,1,""],MsSqlToHiveTransfer:[1,1,1,""],MySqlOperator:[1,1,1,""],MySqlToHiveTransfer:[1,1,1,""],PostgresOperator:[1,1,1,""],PrestoCheckOperator:[1,1,1,""],PrestoIntervalCheckOperator:[1,1,1,""],PrestoValueCheckOperator:[1,1,1,""],PythonOperator:[1,1,1,""],S3KeySensor:[1,1,1,""],S3ToHiveTransfer:[1,1,1,""],ShortCircuitOperator:[1,1,1,""],SimpleHttpOperator:[1,1,1,""],SlackAPIOperator:[1,1,1,""],SlackAPIPostOperator:[1,1,1,""],SqlSensor:[1,1,1,""],TimeSensor:[1,1,1,""],TriggerDagRunOperator:[1,1,1,""],WebHdfsSensor:[1,1,1,""]},"airflow
.operators.BashOperator":{execute:[1,2,1,""]},"airflow.operators.SlackAPIOperator":{construct_api_call_params:[1,2,1,""],execute:[1,2,1,""]},"airflow.operators.docker_operator":{DockerOperator:[1,1,1,""]},"airflow.operators.sensors":{BaseSensorOperator:[1,1,1,""]},airflow:{executors:[1,0,0,"-"],hooks:[1,0,0,"-"],macros:[1,0,0,"-"],models:[1,0,0,"-"],operators:[1,0,0,"-"]}},objnames:{"0":["py","module","Python module"],"1":["py","class","Python class"],"2":["py","method","Python method"],"3":["py","function","Python function"],"4":["py","attribute","Python attribute"]},objtypes:{"0":"py:module","1":"py:class","2":"py:method","3":"py:function","4":"py:attribute"},terms:{"00am":1,"01t23":11,"10gb":1,"10pm":2,"128m":1,"2am":1,"5gb":1,"9ba53fba14eb":[],"__class__":2,"__init__":[1,2],"__name__":[2,8],"_cmd":3,"_host":12,"abstract":[1,2,5,8],"boolean":[1,2],"byte":[1,4],"case":[1,2,4,11,12,14],"class":[0,1,2,6,7,8,14],"default":[],"export":[1,13],"final":1,"float":1,"function":[],"import":
[],"int":1,"long":1,"new":[1,2,4,8,11,12],"return":[0,1,2,8],"short":1,"static":[1,5,8],"super":[1,12],"switch":[3,12],"true":[1,2,4,12,14],"try":[1,2,3],"var":1,"while":[1,2,4,5,7,11,13,14,15],abbrevi:2,abil:2,abl:[3,4,12,13,14],about:[1,2,3,7,10,11,14],abov:[2,3,7,12,15],absenc:2,absolut:2,accept:[1,7],access:[1,2,3,8,12],access_token:15,accord:12,accordingli:2,account:[1,12],acquir:1,across:[3,15],act:[1,2,7,14],action:[1,15],activ:[1,2,4,8,11,12,14],actual:[1,2,4,14],acycl:[1,2,5],add:[1,2,3,7,12],add_task:1,addendum:7,addit:[],addition:12,addprinc:12,adhoc:[],adjust:[3,12],admin:[2,3,4,8,9,12,13],admin_view:8,advanc:14,advantag:12,advis:[1,7],affect:2,after:[1,2,4,11],afterward:1,against:[1,2,4,7,12,14],agari:10,aggregate_db_message_job:2,aggregate_db_message_job_cmd:2,ago:4,agre:7,agreement:[2,7],airbnb:[1,2,10,14],airflow:[],airflow__:3,airflow__core__sql_alchemy_conn:3,airflow_config:3,airflow_conn_:[2,3],airflow_conn_postgres_mast:2,airflow_hom:[3,4,8,13],airflow_set:2,airf
low_tmp_dir:1,airflowignor:1,airflowplugin:8,airflowtestplugin:8,aiyzngq:1,alchemi:3,alert:[1,2,8],all:[0,1,2,3,4,6,7,11,12,14],all_db:6,all_don:[1,2],all_fail:[1,2],all_success:[1,2],alleg:7,alloc:1,allow:[0,1,2,3,4,5,8,9,11,12,13,14,15],allowed_st:1,allowed_team:12,alon:7,along:[1,4,7,14],alongsid:[1,7,11],alreadi:1,alright:14,also:[1,2,3,4,6,7,8,11,12,14],alter:[1,2,3,4,12],altern:[1,11],altert:1,although:12,alwai:[1,2,15],amazon:3,amount:[1,2],analys:15,analyz:2,anand:10,anchor:1,ani:[1,2,3,4,7,14,15],annot:7,announc:10,anomali:8,anoth:[1,2,4,10],ansibl:3,anyon:12,anyth:[2,14],anytim:2,anywher:[1,2],aoen:10,apach:[],api:[],api_call_param:1,api_kei:15,api_param:1,api_vers:1,apikei:15,apiv2:1,appar:1,appear:[1,2,7],append:3,appendix:7,appli:[1,2,6,7,11],applic:[7,8,9,12],appropri:[2,7,12],arbitrari:[2,5],architectur:5,archiv:7,are_dependencies_met:1,are_dependents_don:1,aren:[1,2],arg:1,argument:[],aris:7,around:[1,5,14],arrai:[1,5,8],arthur:10,artwr:10,ascii:1,assert:7,asset:1,as
sign:[],associ:[1,2,7,11],assum:[1,2,3,7,14],asterisk:12,async:6,atom:2,attach:[1,2,7],attach_to:1,attack:12,attempt:[0,1],attent:[1,4],attr:1,attribut:[0,1,2,7],audit:8,auth:[1,12],auth_backend:12,authen:12,authent:[],authmechan:1,author:[1,3,5,7,12,14,15],authorship:7,autocommit:1,automat:[1,2,3],avail:[1,2,3,7,8,9],averag:1,avoid:2,awar:2,azkaban:5,back:[1,4,8],backend:[],backfil:[],background:[1,14],backup:3,bag:1,bag_dag:1,bahchi:[],balanc:2,bar:[1,2],bare:14,base64:4,base:[1,2,3,4,7,8,11],base_executor:[1,8],base_hook:[1,8],base_url:1,basedn:12,baseexecutor:[1,8],basehook:[1,2,8],baseview:8,bash:[1,2,14],bash_1:2,bash_command:[1,2,14],bash_command_to_run:3,bash_queu:14,bashoper:[1,2,14],basi:7,basic:[2,6,8,9,14],bcrypt:12,beauchemin:10,becaus:2,becom:[1,2,5,8,14],beelin:1,been:[1,2,3,7,11,12],befor:[0,1,2,4,12],begin:[2,11],behalf:7,behav:1,behavior:[1,2],behind:[1,3,6,11],belong:12,below:[1,7,8,13,14],benefici:7,best:[1,10],better:14,between:[1,2,14],beyond:[],bin:2,bind:7,bi
nd_password:12,bind_us:12,bit:3,blob:14,block:[1,2,14,15],blog:[7,10],blueprint:8,bodi:1,boilerpl:7,bolk:10,bolkedebruin:10,bonu:1,bool:1,boot:3,both:[1,4],bother:14,boto:1,bottleneck:15,box:[1,3,13],bracket:[7,14],branch:[],branch_fals:2,branchpythonoper:[1,2],broker:[0,3],broker_api:0,broker_url:3,brought:10,brows:[2,4],bruin:10,bucket:[1,3],bucket_kei:1,bucket_nam:1,buffer:1,bug:3,build:[],built:[1,2,3,5,8,9,14],bulk:[4,15],bulk_dump:1,bulk_load:1,bump:2,bundl:1,burn:0,but_this_dag_will_not:2,c12318391:1,cacert:12,cach:12,call:[1,2,3,14],callabl:1,callback:12,can:[1,2,3,4,5,6,8,9,11,12,13,14,15],cannot:[1,2,7,14],capabl:8,capac:2,card:1,care:[0,3],carri:[2,7],cast:1,cat:1,categori:8,caus:[4,7],celeri:[],celery_result_backend:3,celeryexecutor:[1,2,3,4,6,11],cell:1,central:2,cert:1,certain:[1,2],certif:1,cfg:[1,2,3,4,11,12,13,14],chain:2,chang:[3,5,7,11,12,13,14],channel:[1,10],charact:7,character:2,charg:7,charset:1,chart:[],chat:10,check:[0,1,2,4],check_for_bucket:1,check_for_kei
:1,check_for_partit:1,check_for_path:1,check_for_prefix:1,check_for_wildcard_kei:1,check_head:1,check_oper:1,check_output:1,checkoper:1,chef:[3,10],child:[1,2],child_dag:2,child_dag_nam:2,chmod:12,choic:[0,1,14],choos:[1,2,7],chri:10,circuit:1,claim:7,clariti:5,claus:1,clean:1,cleanup:1,clear:[0,1,2,4,11,15],clear_xcom_data:1,cli:[1,2,3,4,11],click:[4,12,14,15],client:[1,6,12],client_id:12,client_secret:12,close:[1,4,12],close_conn:1,closest:1,closest_ds_partit:1,cloud:[3,6],cloudant:[1,6],cloudant_conn_id:1,cloudant_default:1,cloudanthook:1,cmd:1,code:[],col:1,collabor:5,collect:[1,2,8],collect_dag:1,color:1,column:1,com:[1,12,14],combin:[2,7],come:[1,11],comma:[0,1,2],comment:7,commerci:7,commit:[1,10,12],commit_everi:1,common:[2,3,4,7,10,14],commonli:1,compani:8,compar:5,compil:[4,7],complet:[1,2,3],complex:[1,2,5,8],compli:7,complianc:7,complic:14,compon:8,compos:[2,12,14],comprehens:15,comput:7,conceptu:2,concern:2,concur:4,concurr:[0,1,4],concurrency_reach:1,condit:[1,2,6,7],c
onf:[0,1,3],config:[8,12],configur:[],confirm:[0,4,11],confluenc:[],confus:[1,2,4],conjunct:[0,2,13],conn_id:[1,2,3],conn_typ:1,connect:[],connect_timeout:1,connecton:9,consequenti:7,consid:[1,2,3],consider:1,consist:[2,7],conspicu:7,constant:[1,2],constitu:2,constitut:7,constraint:1,constru:7,construct:1,construct_api_call_param:1,construct_ingest_queri:1,constructor:[1,14],contact:[3,7],contain:[1,2,3,7,11,12,14,15],content:[],context:[],continu:[1,5],contract:7,contrib:[1,2,12],contributor:[7,10],contributori:7,control:[1,2,7,15],conveni:[1,2],convent:2,convers:7,cookbook:[],cool:1,coordin:1,copi:[1,3,7,12],copyright:[7,12],core:[],correct:2,correpsond:2,correspond:[1,2,3,4],could:[1,2,14],count:1,counterclaim:7,cours:12,cover:11,cpu:[1,8],craft:1,crawl:[1,11],crawl_for_task:1,creat:[1,2,3,4,8,9,11,12,13,14,15],create_dagrun:1,create_directori:1,creation:1,credenti:12,credit:12,criccomini:10,criteria:[1,2],criterion:1,critic:[1,2],cron:[1,4,11],cross:[2,7,14],crt:12,crypto:[3,4,6
],csv:1,curli:14,current:[1,15],current_st:1,cursor:1,custom:8,customari:7,cwiki:[],cycl:[1,2,14],daemon:[0,1,3],dag:[],dag_1:2,dag_2:2,dag_fold:[1,2],dag_id:[0,1,2,14],dag_md:2,dag_run:1,dag_run_obj:1,dagbag:1,dagbag_report:1,dagfil:2,dagrun:[0,1,4],dagrun_timeout:1,dags_fold:3,dai:[1,11,13,14],daili:[1,4,11],damag:7,dan:[1,10],data:[],data_profiler_filt:12,databas:[0,1,2,3,5,6,8,9,13,14],datafram:1,dataset:1,datasourc:1,date:[0,1,2,7,11,14],date_filter_column:1,datestamp:1,datetim:[1,2,4,11,14],dateutil:1,davydov:10,days_back:1,dbapi_hook:1,dbapihook:1,deadlock:[],deal:4,debug:[0,1,14],decid:[1,3],declar:1,deep:1,def:[1,2,8],default_arg:[1,2,4,14],default_login:12,default_queu:2,defend:7,defer:2,deferred_op:2,defin:[1,2,3,4,5,7,8,11,12,14],definit:[],del:1,delai:[1,4],delegate_to:[],delet:[1,2,11,15],delete_directori:1,delete_fil:1,deliber:[1,7],delimit:[0,1,2],demonstr:2,depend:[],depends_on_past:[0,1,2,4,14],deploi:8,deploy:8,depth:14,deriv:[1,3,7,8],descend:2,describ:[1,2,7],de
scribe_directori:1,descript:[1,2,7],deseri:[0,1,2],deserialize_json:2,design:[2,7,11],desir:[1,14],destin:1,destination_conn_id:1,destination_filepath:1,destination_t:1,detail:[1,2,4,12,15],detect:[1,8],detect_downstream_cycl:1,determin:[1,2,7],dev:[3,6,10],devel:6,devel_hadoop:6,develop:[0,1,12,14],deviat:1,dgaoqozlvea:[],dialog:11,dict:[0,1,2],dictionari:[1,2,14],did:14,die:2,differ:[1,2,7,8,14,15],dimens:1,direct:[1,2,3,5,7],directli:[1,2,4,9],directori:[0,1,2,3,12,14],dirti:13,disabl:1,disclaim:7,discuss:7,disk:1,displai:[3,7,11,14],disregard:[1,14],distinct:1,distribut:[1,2,6,7],do_pickl:0,doc:[1,2,4,14],doc_json:2,doc_md:2,doc_rst:2,doc_yaml:2,docker:[1,10],docker_oper:1,docker_url:1,dockeroper:[1,2],document:[],doe:[0,1,2,3,4,6,7,12,14],does_not_exist:1,doesn:[1,2,14],domain:12,don:[1,2,3,6,7,11],done:[1,2,11,12,14],donot_pickl:[0,1],door:2,dot:[1,2],doubl:[1,3,14],down:[0,2],download:[1,3],downstream:[0,1,2,11],downstream_list:1,downstream_task_id:1,drive:3,driven:8,driver:3
,drop:[1,8],druid:[1,6],druid_datasourc:1,druid_ingest_conn_id:1,druid_ingest_default:1,druid_query_conn_id:1,druid_query_default:1,druidhook:1,dry:0,dry_run:0,ds_add:[1,14],ds_format:1,ds_nodash:1,dubiou:1,dummi:[1,2,8],dummy_1:2,dummy_oper:2,dummy_task:2,dummyoper:[1,2],dump:1,durat:[],dure:[3,15],dynam:[1,2,4,5],each:[1,2,4,7,10,11,14],earlier:14,easi:[2,3,5,8,9,12,15],easier:[1,7],easiest:6,easili:[1,2,5,9,14],echo:[2,14],ecosystem:8,edit:[3,15],editori:7,effect:2,effici:1,egg:3,either:[1,2,3,7,12,13],elabor:7,elaps:1,electron:7,eleg:5,element:3,els:13,elsewher:2,email:[1,2,10,12,14],email_on_failur:[1,14],email_on_retri:[1,14],emailoper:[1,2],emit:[2,14],empti:[1,2],enabl:[],enclos:7,encod:[1,4],encrypt:[1,3,4,6,12],encrypt_s3_log:3,end:[1,7,11],end_dat:[0,1,14],endpoint:1,enforc:[2,4],engin:[1,4,5],enterpris:[],entir:[0,4],entiti:7,entri:[1,11],env:[1,2],environ:[1,2,3,5,6,9,11,14,15],ep_data_pipeline_db_msg_agg:2,equival:[2,6,14],era:4,error:1,especi:[2,4,14],essenti:1,etc:[1
,2,3,12],eval:1,evalu:[1,2,4,11,14],evaluate_trigger_rul:1,even:[1,2,7,9],event:[2,7,14],eventlet:0,ever:1,everi:[1,2,11,12,14],everyon:14,everyth:[14,15],exactli:[1,15],exampl:[],example1:13,example_bash_oper:13,example_dag:[2,14],example_team_1:12,example_team_2:12,exce:1,except:[1,2,4,7,14],exchang:[2,5],exclud:7,exclus:[7,11],execut:[0,1,2,3,4,5,7,11,14],execution_d:[0,1,2,14],execution_delta:1,execution_timeout:[1,2],exercis:7,exhaust:3,exist:[0,1,2,3,4,14],exit:[0,12],expect:[1,2,4,5],expir:1,explain:1,explan:14,explicit:5,explicit_op:2,explicitli:[1,2,7,12,14],expos:[1,2,3,8,12],express:[1,4,7,11],ext:1,extend:5,extens:[1,5],extern:[],external_dag_id:1,external_task_id:1,external_trigg:1,externaltasksensor:1,extra:[],extra_dejson:1,extra_opt:1,fact:2,factori:2,fail:[0,1,2,3,4,11,14,15],failur:[1,3,4,7],fairli:13,fall:10,fals:[0,1,3,11,14],familiar:[2,14],faq:[],fast:13,fault:2,featur:[1,2,3,6,8,14,15],feb:12,fee:7,fernet:4,fernet_kei:4,fetch:3,few:[1,3,9,13,14],field:[1,7],fi
eld_dict:1,fifti:7,figur:[1,2,11],file:[],filenam:[1,2],filepath:1,filestatu:1,filesystem:3,fill:[1,2,9,12],filter:[0,1,2,12],filter_by_own:12,find:[1,3,4,14,15],fine:14,finish:1,fire:[1,2,3],first:[0,1,2,3,4,10,11,13,14],fit:[1,5,7],fix:[1,11],flag:[1,11],flag_upstream_fail:1,flask:[0,8,9],flask_admin:8,flask_blueprint:8,flask_login:12,flat:1,flavor:8,flexibl:[1,14],flower:[0,3],folder:[1,3,8,11,13,14],follow:[1,2,3,5,7,12,14],foo:[1,2],foo_dag_id:4,foobar:3,forc:[0,1],force_pul:1,foreground:0,foreign:1,form:[],format:[1,2,3,7],forward:[1,4],found:[1,3,12],foundat:[7,10,13],four:2,framework:[3,8],free:[2,7],frm:1,from:[0,1,2,3,4,5,7,8,9,10,11,12,13,14,15],ftp:1,ftp_conn_id:1,ftp_default:1,ftphook:1,full:[1,10],full_filepath:1,fulli:[4,12],functiontyp:1,fundament:14,further:[13,14],futur:[1,11],galaxi:8,gantt:[],garag:2,gate:12,gc_base_hook:[],gcloud:[],gcp_api:6,gender:1,gener:[0,1,2,5,7,8,12,14,15],generictransf:1,get:[],get_active_run:[],get_bucket:1,get_conn:1,get_connect:2,get_
cursor:1,get_dag:1,get_databas:1,get_direct_rel:1,get_first:1,get_flat_rel:1,get_kei:1,get_metastore_cli:1,get_pandas_df:1,get_partit:1,get_partitions_by_filt:1,get_record:1,get_tabl:1,get_task_inst:1,get_template_context:1,get_template_env:1,get_wildcard_kei:1,gevent:0,ghe:[],ghe_oauth:12,ghost:1,git:3,github:[],github_enterpris:12,github_enterprise_auth:12,githubusercont:1,gitignor:1,gitter:10,give:[2,7],given:[1,2],glanc:15,global:[1,2,4],goal:14,goe:[1,4,14],good:2,goodby:2,goodwil:7,googl:[3,6],google_cloud_storage_conn_id:[],google_cloud_storage_default:[],googlecloudbasehook:[],googlecloudstoragehook:[],govern:7,grab:1,grade:3,grai:1,grant:7,graph:[],great:3,greater:1,green:1,grossli:7,group:[1,3,12],gunicorn:[0,6],hack:1,had:[1,4],hadoop:[6,10,12],hadoop_dependency_coordin:1,hand:[12,13],handi:10,handl:[1,2,3],happen:[2,11,14],hard:[1,2],harmless:7,has_dag:1,hash:[1,12],hasn:2,have:[1,2,3,4,6,7,8,9,11,12,14],haven:[1,2,14],hdf:[1,2,6],hdfs_conn_id:1,hdfs_default:1,hdfs_path:
1,hdfscli:1,hdfssensor:1,head:[13,14],header:1,heartbeat:[1,2],heavier:1,hello:[2,8],help:[0,7,8,12],here:[1,2,3,4,6,8,10,11,13,14,15],herebi:7,herein:7,hidden:15,hierarchi:[1,14],high:1,highchart:9,higher:1,highli:3,highlight:14,hipchat:1,hipchat_oper:1,hipchatapioper:1,hipchatapisendroomnotificationoper:1,hit:[1,2],hive2mysqloper:1,hive2sambaoper:[1,6],hive:[1,2,3,6,8,12],hive_cli_conn_id:1,hive_cli_default:1,hive_cli_param:1,hive_conn_id:1,hive_t:1,hiveclihook:1,hiveconf:1,hiveconf_jinja_transl:1,hivemetastorehook:1,hiveoper:[1,2,3,12],hivepartitionsensor:[1,2],hivesensoroper:1,hiveserver2:1,hiveserver2_conn_id:1,hiveserver2_default:1,hiveserver2hook:1,hivetodruidtransf:1,hivetomysqltransf:1,hold:7,home:13,homogen:3,hope:11,horribl:14,host:[1,3,12],hostnam:[0,1,2,3],hour:[1,2,4,11],hourli:[1,4,11],hous:2,how:[1,2,3,4,7,8,11,14,15],howev:[2,7,12],hql:1,html:[1,8,14],html_content:1,http:[1,2,7,8,12,14],http_conn_id:1,http_default:1,httphook:1,httpoper:2,httpsensor:1,human:1,icon:1,
icon_url:1,idea:[],idempot:1,identif:7,identifi:[1,7,14,15],ignor:[0,1],ignore_depend:[0,1],ignore_depends_on_past:[0,1],ignore_first_depends_on_past:[0,1],iii:7,imag:1,immedi:[1,2],imper:3,imperson:12,implement:[1,2],impli:[1,7],improv:7,impyla:1,inabl:7,inact:4,inc:7,incident:7,includ:[0,1,2,3,7,11,14],include_adhoc:[0,1],include_downstream:1,include_exampl:1,include_prior_d:1,include_queu:1,include_upstream:1,inclus:7,incorpor:7,incub:[],incur:7,indemn:7,indemnifi:7,indentifi:0,independ:[1,2],index:1,indic:[1,2,7],indirect:7,individu:[1,7,11,14],infer:[1,2],inferred_op:2,infil:1,infin:5,info:1,inform:[1,2,3,7,11,12,14],infrastructur:1,infring:7,ingest:1,inherit:[1,14],init:3,initctl:3,initdb:[0,3,13],initi:[0,3,12,13],initialis:1,inject:8,input:1,input_format:1,insecur:12,insecurecli:1,insert:[1,2],insert_row:1,insid:[1,2],inspect:[11,13],instal:[],instanc:[],instanti:[],instati:14,instead:[0,1,4],insteadnhttp:1,institut:7,instruct:2,insur:1,integr:[],integrate_plugin:1,intention
:7,inter:2,interact:[1,2,3,8,9],interest:[1,14],intergr:8,intern:0,interpret:[1,2],interv:1,intervalcheckoper:1,introduc:4,intuit:14,invari:[1,2],invok:1,ipython:2,irrevoc:7,is_paus:1,is_prematur:1,is_queueabl:1,is_runn:1,ish:1,isn:[1,2,4],isoformat:1,issu:[1,2,5,7,10],item:2,iter:1,itself:[2,12,14],januari:[7,11],java:1,jdbc:[1,6],jeremiah:10,jinja2:1,jinja:[],jinja_env:1,jira:[],jlowin:10,job:[0,1,2,3,4,11,13,15],job_id:1,jobtrack:1,join:[2,10],json:[0,1,2],jupyt:2,just:[0,1,2,4,8,14],kadmin:12,kapil:3,keep:[],kei:[0,1,2,3,4,12,15],kerber:[6,12],kerbero:[],key_fil:1,key_path:[],keytab:[0,12],keyword:1,kick:[3,11],kill:[1,2],kill_zombi:1,kind:7,know:[3,14],knowledg:8,known:6,known_host:1,kwarg:1,kwd:1,lab:1,label:[1,9],lai:13,lambda:[1,2],land:[1,8],languag:[7,14],larg:1,larger:1,last:1,late:15,later:9,latest:[0,1,6,14],latest_d:1,latest_execution_d:1,launch:12,law:7,lawsuit:7,layout:14,ldap3:12,ldap:[],ldap_auth:12,ldap_ca:12,lead:[1,2],lean:5,least:[1,2,7],leav:1,led:2,left:[2,12
],legaci:4,legal:7,len:1,leo:10,less:1,lesson:10,let:[0,1,2,3,9,14,15],level:[1,2,5,11,12],leverag:[1,2,14],liabil:7,liabl:7,lib:[1,2,3],libffi:3,librari:[1,2,3,5,8,14],libz:2,licens:[],licensor:7,life:2,light:[1,2],lighter:1,lightweight:2,like:[1,2,3,6,8,12,14],limit:[],link:[],linkedin:[],list:[0,1,2,3,4,6,7,8,10,12,14,15],list_dag:[0,4,14],list_directori:1,list_kei:1,list_prefix:1,list_task:[0,4,14],listen:2,lit:1,liter:1,litig:7,live:1,load:[1,2],load_fil:1,load_from_hdf:1,load_str:1,local:[0,1,2,3,12,14],local_executor:1,local_full_path_or_buff:1,local_infil:1,local_port:1,localexecutor:[0,1,2,3,11],localhost:2,locat:[0,1,2,3,12,14],lock:1,lock_for_upd:1,log:[],log_fil:0,loggingmixin:1,logic:[1,2,14],login:[1,2,3,12],logist:8,look:[0,1,2,5,8,10,11,14],loss:7,lot:11,lowin:10,machin:[2,3,12],made:[1,2,7],mai:[1,2,3,4,7,11,14],mail:[7,10],maill:[],main:[1,2,4,8],main_dag:2,maintain:[1,5,10],make:[1,2,3,4,5,7,12,14,15],malfunct:7,malik:3,man:6,manag:[],mani:[0,1,2,4,8,14,15],manual
:[1,2,3,11],map:1,mapr:1,mark:[0,1,2,7,11],mark_success:[0,1,11],markdown:2,markup:1,master:[1,2,14],match:[0,1,2],matter:1,max:[1,10],max_active_run:[1,4],max_partit:1,maxim:10,maxime_beauchemin:0,maximum:1,mayb:2,mean:[1,2,3,4,7,12,14],meaning:1,meant:1,mechan:[1,7,12],media:7,medium:7,meet:[1,7],mem_limit:1,member:12,memberof:12,memori:1,menu:[],menu_link:8,menulink:8,merchant:7,mere:7,merg:14,mesosexecutor:[3,11],messag:[1,2,5],message_format:1,met:[1,3,4,11],metadata:[],metastor:1,metastore_conn_id:1,metastore_default:1,metastore_mysql:1,metastorepartitionsensor:1,method:[1,2],metric:[1,8],metric_spec:1,metrics_threshold:1,microsoft:[1,6],midnight:[4,11],might:[1,2,4,11,14],migrat:12,mime:[],mime_typ:[],min:4,mind:[],minimum:[1,6,11],minut:[2,4,11,14],mismatch:2,miss:[1,2],mistercrunch:10,mix:14,mkdir:2,mlsd:1,mode:[0,1,14],modif:7,modifi:[1,2,3,7,9],modul:[],modular:5,monitor:[2,3,5,8,11,15],monospac:2,month:[4,11],monthli:[4,11],more:[1,2,3,5,7,8,11,12,14,15],morn:11,most:[1,
2,12,14,15],mostli:[1,5,11],mount:[1,3],move:[1,2,4,5],msg:1,mssql:[1,6],mssql_conn_id:1,mssql_default:1,mssqlhook:1,mssqloper:1,mssqltohivetransf:1,much:[1,2,11],multi:[],multipart:1,multipart_byt:1,multipl:[1,2],multipli:1,multiprocess:1,must:[1,2,3,4,7,12,14],mutat:2,my_conn_str:3,my_dag1:2,my_dag2:2,my_dag:2,my_databas:1,my_funct:2,my_param:14,my_tabl:1,my_useful_packag:2,myarg:1,mypackag:12,mys3conn:3,mysql:[1,2,3,6],mysql_conn_id:1,mysql_default:1,mysql_postoper:1,mysql_preoper:1,mysql_tabl:1,mysqldb:1,mysqlhook:1,mysqloper:[1,3],mysqltohivetransf:1,name:[1,2,3,7,8,12],namespac:[1,8],nav:12,navig:12,ndfor:14,necessarili:7,need:[1,2,3,4,5,6,8,11,12,13,14],neg:[1,11],neglig:7,neighbour:1,nerd:[],nest:[1,14],network:[1,12],network_mod:1,never:[0,1,4],new_user_email:12,new_user_nam:12,newobject:1,next:[],nhere:1,nice:2,night:2,nlst:1,no_confirm:0,no_host_key_check:1,node:[0,1,2],non:[1,3,7],none:[1,2,8,11],noram:12,norandkei:12,normal:[1,2,7],notat:1,note:[],notebook:2,noth:[1,7],
notic:[2,7,14],notif:1,notifi:1,notwithstand:7,nov:10,now:[1,2,3,4,12,14],nuanc:2,num:1,num_run:0,num_shard:1,number:[0,1,2,3,5],o2snqeoz7:[],oauth2:12,oauth_callback_rout:12,oauth_key_from_github_enterpris:12,oauth_secret_from_github_enterpris:12,obj:1,object:[0,1,2,3,4,7,8,11,14],objectclass:12,oblig:7,observ:1,obtain:7,occur:[1,2],occurr:1,octet:[],off:[1,3,11],offer:[7,8],offici:[1,10],offset:1,often:[1,2],on_failure_callback:1,on_kil:1,on_retry_callback:1,on_success_callback:1,onboard:4,onc:[1,2,3,4,11,14],one_fail:[1,2],one_success:[1,2],onli:[0,1,2,3,4,6,7,11,12,13],only_fail:0,only_if_upd:1,only_run:0,oozi:5,op1:2,op2:2,op3:2,op4:2,op_arg:1,op_kwarg:1,open:[1,2,4,10,12],oper:[],oppos:[1,2],optim:1,option:[],orchestr:[1,5],order:[1,2,3,11,12],org:[7,8,10,14],organ:[2,8],organiz:12,origin:7,orm:1,other:[1,2,3,5,7,9,11,14],otherwis:[1,7,14],our:14,out:[],outag:8,outgrow:13,outlier:15,output:[1,14],output_encod:1,output_format:1,outsid:11,outstand:7,over:[0,1,2,3,13,15],overlap:
15,overrid:[0,1,14],overridden:[1,4],overview:15,overwhelm:2,overwrit:1,own:[],owner:[1,2,7,12,14],ownership:7,oyp49mbwh60:[],p12:[],packag:[],package1:2,page:[2,4,7,10,15],pai:[1,4],pair:[1,15],panda:1,parallel:[1,2,3,13],param:[0,1,14],paramet:[0,1,2,3,4,9,14],parameter:[2,5,9],parent:2,parent_dag:[1,2],parent_dag_nam:2,pars:[1,4,8,14],part:[1,2,7,9,12],parti:7,particular:7,partit:[1,2],partition_nam:1,partli:4,pass:[1,2,4,8,14],pass_valu:1,passwd:15,password:[],password_auth:12,passwordus:12,past:[1,2,4,11,15],patent:7,path2:1,path:[1,2,3],patrick:10,patrickleotardif:10,pattern:[1,2],paus:[0,1],payload:1,pem:1,peopl:[1,8,14],percent:7,perfect:2,perform:[0,1,2,3,5,7,15],perhap:[2,15],period:[1,2,4,11,14],permiss:[7,12],perpetu:7,persist:[1,11],perspect:2,pertain:7,phase:8,pick:[2,3,9],pickl:[0,1,2,3],picklabl:1,pickle_id:1,pickleabl:1,pid:[0,13],pig:[1,2],piggi:8,pigoper:1,pin_100:1,pip:[2,6,13],pipelin:[],pitfal:10,place:[1,2,7,14],placehold:1,plain:[1,3],plan:4,platform:[2,5,6],
pleas:[2,3,10,12,14],plug:8,plugin:[],pluginexecutor:8,pluginhook:8,pluginoper:8,plugins_manag:8,png:1,point:[1,2,3,4,13,14],poke:1,poke_interv:1,polymorph:1,pool:[],pool_ful:1,popen:1,popul:14,port:[0,1,3,12,13],posit:[0,1],possibl:[0,1,2,3,7,12,13,14,15],post:[1,10],post_execut:1,postgr:[1,2,3,6],postgres_conn_id:1,postgres_default:1,postgres_mast:2,postgreshook:1,postgresoper:1,potenti:2,power:[2,5,7,14],practic:[1,10],pre:4,pre_execut:1,preced:[1,3,14],prefer:[7,11,13],prefix:[1,2,3],preoper:1,prepar:[2,7],prepare_templ:1,prepped_request:1,present:1,preset:11,presto:1,presto_conn_id:1,presto_default:1,prestocheckoper:1,prestohook:1,prestointervalcheckoper:1,prestotomysqloper:2,prestovaluecheckoper:1,pretti:14,prevent:[1,2,12],previou:[0,1,2,4,14],previous:2,primari:1,primit:1,princip:[0,2,12],print:[1,2,7,14],print_dat:14,prior:1,priorit:2,prioriti:[1,2],priority_weight:[1,2,14],privaci:7,probabl:[3,4,12],problem:2,problemat:2,proce:[1,4],process:[0,1,2,3,11,14],process_fil:1,pr
oduct:[1,2,3,5,7,9,11,14],profil:[],program:10,programmat:5,progress:[1,5,14],project:[],promin:7,prompt:0,propag:[1,2],propaget:1,proper:14,properli:[3,4],properti:[1,2,4,12],provid:[1,2,3,7,9,14,15],provide_context:[1,2],proxy_us:[1,12],proxyus:12,psycopg2:3,ptarget:1,publicli:7,publish:1,pull:[1,2],pull_funct:2,puls:[],puppet:[3,10],pure:2,purpl:1,purpos:[7,14],push:[1,2,8],push_funct:2,pushing_task:2,put:[1,2,8],pwd:2,pyhiv:1,pypi:[6,13],python:[1,2,3,5,6,8,12,14],python_1:2,python_cal:[1,2],python_oper:1,pythonhost:8,pythonoper:[1,2],pythonpath:[3,12],qualif:12,qualifi:12,qualiti:[1,2],qubol:6,queri:[],queu:[1,2,4],queue:[],quick:[],quickli:[9,13,14,15],quit:4,r39132:10,rabbitmq:[3,6],rais:[1,14],ran:0,randkei:12,random:1,randomli:2,rang:[0,1,11,14],rather:2,ratio:1,raw:1,reach:[1,2,3,4],reactiv:4,read:[1,3,4,12,14],readabl:[1,7,12],readi:[1,5,13],readm:3,readthedoc:14,ready_for_retri:1,real:3,realli:[1,12,14],realm:12,reason:[4,7,14],reboot:2,rebuild:0,recap:[],receiv:[1,2,7,1
1],recent:[1,2],recip:10,recipi:7,recogn:1,recommend:[1,3,4,7],record:[0,1,2,14],recreat:1,recurs:1,red:1,redhat:3,redi:3,redirect:0,redistribut:7,redund:14,referenc:[2,14],reflect:[2,8,14],refrain:2,refresh:1,refresh_from_db:1,regard:[1,7],regardless:[0,1],regex:[0,1],regex_kei:1,regexp:0,regist:[8,9,12],regular:[1,11],reinit_frequ:12,rel:[1,11,14],relat:[1,2,3,4,6],relationship:2,relativedelta:1,relev:4,reli:[1,12],reliabl:1,remain:[7,12],rememb:[1,14],remot:[1,3,11],remote_base_log_fold:3,remote_full_path:1,remote_host:1,remote_log_conn_id:3,remote_port:1,remov:1,render:[0,1,2,8,14],render_templ:1,render_template_from_field:1,renew:[0,12],repeat:2,repl:12,replac:[1,7,12],report:1,repositori:[3,14],repres:[1,2,4,7,11],represent:[1,15],reproduc:7,reproduct:7,request:[0,1,2],requir:[1,2,3,6,7,12],resembl:2,reset:0,resetdb:0,resid:2,resolv:1,resourc:[],respect:[0,1,2,14],respons:[1,7],response_check:1,rest:1,restart:[2,3,4],restrict:12,restructuredtext:2,result:[1,2,7,9,14],retain:7,
retri:[1,2,14],retriev:[1,2],retrieve_fil:1,retry_delai:[1,14],return_valu:1,reus:8,revis:7,riccomini:10,rich:[0,2,5,15],rid:1,right:[1,2,7,9],rightmost:2,risk:7,role:3,roll:[],room:1,room_id:1,root:[1,2],root_dag:1,round:[1,4],rout:12,routin:[1,2],row:[1,2],royalti:7,rule:[],run:[],run_a:1,run_and_check:1,run_as_own:[1,12],run_cli:1,run_id:[0,1,11],runme_0:13,runnabl:[1,2],rwc:12,s3_conn_id:1,s3_default:1,s3_kei:1,s3connect:1,s3filetransferoper:2,s3hook:[1,3],s3keysensor:[1,6],s3prefixsensor:6,s3tohivetransf:1,safe_mod:1,sai:[1,2,4,11],said:2,samaccountnam:12,samba:[1,6],samba_conn_id:1,samba_default:1,same:[1,2,3,7,9,12],sampl:3,sandbox:3,save:[3,4,9,14],scalabl:5,scan:[1,2],scene:[6,11],schedul:[],schedule_interv:[1,2,4,11,14],scheduler_interv:4,scheduler_run:3,schema:[1,2,3],scratch:14,screen:14,screenshot:[],script:[],script_begin_tag:1,search:[1,2],search_path:3,search_scop:12,second:[1,2,4,14],secret:[12,15],section:[1,2,3,4,7,12,13,14],secur:[],see:[1,2,3,4,7,11,12,13,14,15]
,seen:[3,15],segment:1,select:[1,2,9,11,12],self:[1,8],sell:7,semicolon:1,send:[0,1,2,10],send_room_notif:1,sender:1,sensor:[1,2,4],sensor_queu:2,sent:[1,2,3,7],separ:[2,7,14],sequenti:[1,11,13],sequentialexecutor:[1,2,13],serial:[0,1],serv:[0,3,14],serve_log:[0,3],server:[0,1,3,8,12,13,14],servic:[0,1,2,3,6,7,11,12],service_account:[],session:[1,12],set:[],set_depend:[1,14],set_downstream:[1,2,14],set_the_password:12,set_upstream:[1,2,14],setgid:3,setuid:3,setup:[3,12],sever:2,shall:7,shape:11,share:[1,2,3,7,8,9],shell:12,ship:[0,1],ship_dag:0,shop:7,shortcircuitoper:1,shortcut:[3,15],shortest:1,should:[1,2,3,4,7,8,11,13,14],shouldn:[1,2],show:[0,1,2,4,8,15],shown:1,sid:10,siddharth:10,side:[1,3],signal:1,similar:[2,3,5],simpl:[1,2,8,9,14],simplehttpoper:1,simpler:14,simplest:12,simpli:[1,2,8,11,12,14],simplifi:1,simul:14,simultan:2,sinc:[1,3,13,14],sing:1,singl:[0,1,2,14],site:12,size:[1,2],skew:8,skip:[0,1,2],sla:[],sla_miss_callback:1,slack:[1,6],slack_oper:1,slackapioper:1,slac
kapipostoper:[1,6],slackoper:2,slave:3,sleep:14,slightli:5,slot:[1,2],slowli:5,slug:12,small:[1,2],smaller:1,smallish:1,snap:5,sock:1,soft_fail:1,softwar:[7,10],sole:7,solut:5,some:[1,2,3,4,11,14,15],somehow:3,someth:[2,12,14],sometim:[1,2,14],somewhat:14,somewher:13,soon:[1,2,3,4,11],sort:2,sound:[2,14],sourc:[1,2,7,9,10,15],source_conn_id:1,space:5,span:15,spark:[2,5],spawn:3,special:[1,2,4,7],specif:[0,1,2,3,4,7,11,14,15],specifi:[0,1,2,3,4,5,11,12,14],specific_valu:1,spent:15,spring:10,sql:[1,2,3,6,9],sql_alchemy_conn:3,sql_alchemy_conn_cmd:3,sqlachemi:1,sqlalchemi:[1,2,3],sqlite:[1,13],sqlitehook:1,sqloper:2,sqlsensor:1,squeez:1,sscursor:1,ssh:[1,12],ssh_default:1,ssh_hook:1,sshexecuteoper:1,sshhook:1,sshpass:1,ssl:1,sslcert:1,sslmode:1,stabl:6,stack:[6,8],stage:[1,2],stai:11,stamp:11,stand:2,standalon:2,standard:[1,2],stanza:3,start:[],start_dat:[0,1,2,4,11,14],stat:1,state:[0,1,2,4,7,11,14],state_d:1,statement:[1,2,7],static_babynam:1,static_babynames_partit:1,static_fold:8,s
tatic_path:1,static_url_path:8,statu:[0,1,2,3,4,7,11,13,14,15],stderr:0,stdout:[0,1,14],step:[1,3,14,15],still:[1,4],sting:1,stop:[1,3],stoppag:7,storag:3,store:[1,2,3,8,12,13],store_fil:1,storm:5,str:[1,8,11],straightforward:13,stream:[1,5],string:[0,1,2,3,14],string_data:1,structur:[1,5,14],style:1,sub:[0,1],sub_dag:[1,2],sub_part:1,subcommand:[1,3,11,14],subdag:[],subdagoper:2,subdir:0,subdirectori:2,subject:[1,7],sublicens:7,submiss:[7,11],submit:7,subpackag:[6,12],subpartit:1,subprocess:[1,11],subscrib:[2,10],subsect:[0,11],subsequ:[0,7,11],subset:1,subtl:2,subtract:1,subtre:12,succe:[1,2],succeed:[0,1,2,4,15],success:[0,1,2,4,11,14],successfulli:[1,2,14],suit:5,sum:2,summit:10,sundai:11,super_us:12,supersed:7,superus:12,superuser_filt:12,suppli:[3,12],support:[0,1,3,6,7,12],sure:[1,2,3,4,12,14],surfac:14,surgeri:5,sync:[0,1,3,11],sync_to_db:1,synchron:[1,3],syntax:7,sysconfig:3,system:[1,2,3,7],tab:[1,12],tabl:[1,2,8],table_exist:1,table_nam:[1,2],tablenam:1,tag:4,take:[1,2,3,
10,12,13,14,15],talk:[1,10,14],tardif:10,target:[1,3,11],target_field:1,target_partition_s:1,target_tim:1,task:[],task_id:[0,1,2,11,14],task_inst:[1,2],task_instance_key_str:1,task_or_task_list:1,task_param:0,task_regex:[0,1],task_stat:0,taskinst:[1,2],tbl:1,team:[1,12],tech:[],tell:[0,1],templat:[],template_fold:8,template_searchpath:[1,14],templated_command:14,templates_dict:1,templates_ext:1,temporari:[1,2],tenanc:[],tenant:12,term:7,termin:[2,7],test:[],test_env:2,test_hql:1,test_mod:1,test_plugin:8,testabl:5,testview:8,text:[1,3,7,15],textfil:1,than:[1,2,5,14],thei:[0,1,2,3,5,6,8,9,11,14],them:[0,1,2,8,11,14],themselv:2,theori:[4,7],thereof:7,thi:[0,1,2,3,4,5,7,8,11,12,13,14,15],thin:1,thing:[1,2,4,14],think:[5,14],third:7,this_dag_will_be_discov:2,those:[1,2,3,7,12],though:[2,5],thousand:2,thread:1,three:[2,3],threshold:8,thrift:1,through:[1,2,6,7,11,13,14],throughout:1,ticket:[0,12],tighten:12,time:[1,2,3,4,8,11,12,14,15],timedelta:[1,2,4,11,14],timedeltasensor:[1,4],timefram
:1,timeout:[0,1,2],timesensor:1,timestamp:[1,2,11],tip:[2,10],titl:7,tls_ca_cert:1,tls_client_cert:1,tls_client_kei:1,tls_hostnam:1,tls_ssl_version:1,tmp:[1,2],tmp_dir:1,tmp_file:1,todai:1,togeth:[1,2],token:1,toler:[1,2],tomorrow:1,tomorrow_d:1,tomorrow_ds_nodash:1,too:[1,2,11],tool:[1,2,6,8],toolbox:8,top:[],topic:3,tornado:0,tort:7,tour:[13,14],track:[2,7,14],tracker:1,trade:7,trademark:7,tradit:1,tradition:2,train:7,transact:1,transfer:[1,2,7],transform:7,translat:[1,7],transpar:15,travers:1,treat:[1,2],tree:[],tree_view:1,tri:[1,12],trick:10,trigger:[],trigger_dag:[0,11],trigger_dag_id:1,trigger_rul:[1,2],triggerdagrunoper:[1,10],triggerrul:1,troubl:6,troubleshoot:[5,15],truncat:1,truth:1,ts_dim:1,ts_nodash:1,tsv:1,tty:1,tunnel:[1,12],tupl:1,turn:[2,12],tutori:[],twice:1,two:[1,2,14],type:[0,1,2,4,7,9,12],typic:[1,2],ubuntu:3,uid:12,ultim:[1,14],unassign:2,unavail:3,unblock:11,undefin:12,under:[1,2,3,7,8,10],underli:8,underscor:3,understand:[1,4,8,14,15],undesir:11,unexpect:2,u
nexpectedli:2,union:7,uniqu:[1,14],unit:[3,5,12],unittest:1,unix:1,unless:[4,7],unpack:[1,2],unpaus:0,unset:1,unsound:[1,2],unsuccess:1,until:[1,2,3],up_for_retri:1,updat:[1,2,4,12],upgrad:0,upgradedb:0,upload:1,upon:[2,3,13],upstream:[0,1,2,4,11],upstream_fail:[1,2],upstream_list:1,upstream_task_id:1,uri:[1,2,3,12],url:[1,3,8,9,12],usag:[0,14],use_beelin:[1,12],user:[0,1,2,3,5,6,12,14],user_defined_macro:1,user_filt:12,user_name_attr:12,usernam:[1,3,12],usr:3,usual:[2,3],utf8:1,utf:1,util:[1,5,13],uuid:1,val:[0,1],valid:[],valu:[0,1,2,3,4,12,14,15],valuecheckoper:1,variabl:[],vast:1,verbal:7,verbos:[1,14],veri:[0,1,2,4,10,13,14],verifi:[2,4],version:[0,1,2,5,6,7],vertia:1,vertica:[1,6],vertica_conn_id:1,vertica_default:1,verticahook:1,verticaoper:1,verticaql:1,verticatohivetransf:1,via:[1,2,12],video:[1,10],view:[],virtualenv:2,visibl:[1,2],visual:[5,8,9,14,15],vocabulari:2,volum:1,wai:[1,2,3,4,6,8,11,12,14,15],wait:[0,1,2],wait_for_downstream:[1,4],wait_for_empty_queu:2,walk:14,wa
nt:[1,2,3,4,8,11,12,14],warranti:7,wasn:4,wasnt:1,watch:[1,3,4],weapon:9,web:[],webhdfs_conn_id:1,webhdfs_default:1,webhdfshook:1,webhdfssensor:1,webserv:[0,3,4,12,13,14],week:11,weekli:11,weight:1,well:[1,2,3,4,11],were:1,what:[],whatev:[2,3,6],when:[1,2,3,4,5,8,9,11,12,14],whenev:1,where:[1,2,3,4,7,11,12,14,15],wherev:[1,7],whether:[1,7,11],which:[0,1,2,3,4,7,11,12,13,14],whitelist:12,who:8,whole:[1,2,7],whom:7,whose:11,why:[],wiedmer:10,wiki:10,wildcard:1,wildcard_kei:1,wildcard_match:1,win:1,window:11,winter:10,wire:2,within:[0,1,2,3,4,7],without:[0,1,2,7,11,12],won:[1,4,6],word:[2,3,11,15],work:[0,1,2,3,4,5,7,8,9,11,12,13],worker:[0,1,2,3,4,5,6,8,12,14],worker_timeout:0,workerclass:0,workflow:[],workload:8,worldwid:7,worri:14,worth:1,would:[1,2,3,4,5,11,12,14],wouldn:[1,4],wrap:14,wrapper:1,write:[3,5,7,8,9,14],written:[1,7,14],wrong:14,www:[1,7],x01:1,xcom:[],xcom_al:1,xcom_pul:[1,2],xcom_push:[1,2],xcom_return_kei:1,xml:12,xst:12,yaml:2,year:[1,11],yearli:11,yellow:1,yesterda
i:1,yesterday_d:1,yesterday_ds_nodash:1,yet:[1,2,12,15],you:[1,2,3,4,5,6,7,8,9,11,12,13,14,15],your:[],yourself:1,youtu:[],youtub:1,yum:6,yyyi:[0,1,2],yyyymmdd:1,zero:1,zip:[1,2],zip_dag:2,zip_dag_cont:2,zoom:2},titles:["Command Line Interface","API Reference","Concepts","Configuration","FAQ","Apache Airflow (incubating) Documentation","Installation","License","Plugins","Data Profiling","Project","Scheduling & Triggers","Security","Quick Start","Tutorial","UI / Screenshots"],titleterms:{"default":[1,2,14],"function":2,"import":14,addit:2,adhoc:9,airflow:[5,6,8],apach:5,api:1,argument:[2,14],assign:2,authent:12,backend:3,backfil:14,baseoper:1,basesensoroper:1,beyond:5,bitshift:2,branch:2,build:8,celeri:3,chart:[9,15],cluster:2,code:15,command:[0,14],committ:10,commun:[1,3],composit:2,concept:2,configur:3,connect:[2,3],content:5,context:[2,15],contribut:[1,3],core:2,dag:[2,11,14,15],data:9,definit:14,depend:14,document:[2,5],durat:15,enabl:12,enterpris:12,exampl:[8,14],executor:1,
extern:11,extra:6,faq:4,file:14,form:9,gantt:15,get:6,ghe:12,github:12,graph:15,histori:10,hook:[1,2],horizon:5,idea:2,incub:5,instal:6,instanc:[2,15],instanti:14,integr:3,interfac:[0,8],jinja:[2,14],keep:11,kerbero:12,ldap:12,licens:7,limit:12,line:[0,14],link:10,log:3,macro:1,manag:2,menu:15,meso:3,metadata:14,mind:11,model:1,modul:14,multi:12,next:[13,14],note:2,oper:[1,2],option:3,out:3,own:12,packag:[2,6],password:12,pipelin:14,plugin:8,polici:2,pool:2,principl:5,profil:9,project:10,queri:9,queue:2,quick:13,recap:14,refer:1,resourc:10,roadmap:10,roll:12,rule:2,run:[11,14],scale:3,schedul:11,scope:2,screenshot:[9,15],script:14,secur:12,set:[3,12,14],sla:2,start:13,subdag:2,systemd:3,task:[2,14,15],templat:[2,14],tenanc:12,test:14,top:8,tree:15,trigger:[2,11],tutori:14,undead:2,upstart:3,valid:14,variabl:[1,2,15],view:15,web:12,what:[8,13,14],why:8,workflow:2,xcom:2,your:12,zombi:2}})
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9e19165c/security.html
----------------------------------------------------------------------
diff --git a/security.html b/security.html
new file mode 100644
index 0000000..7ca3bf7
--- /dev/null
+++ b/security.html
@@ -0,0 +1,436 @@
+
+
+<!DOCTYPE html>
+<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
+<head>
+ <meta charset="utf-8">
+
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
+
+ <title>Security — Airflow Documentation</title>
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+
+
+
+
+
+ <link rel="top" title="Airflow Documentation" href="index.html"/>
+ <link rel="next" title="FAQ" href="faq.html"/>
+ <link rel="prev" title="Plugins" href="plugins.html"/>
+
+
+ <script src="_static/js/modernizr.min.js"></script>
+
+</head>
+
+<body class="wy-body-for-nav" role="document">
+
+ <div class="wy-grid-for-nav">
+
+
+ <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+ <div class="wy-side-scroll">
+ <div class="wy-side-nav-search">
+
+
+
+ <a href="index.html" class="icon icon-home"> Airflow
+
+
+
+ </a>
+
+
+
+
+
+
+
+<div role="search">
+ <form id="rtd-search-form" class="wy-form" action="search.html" method="get">
+ <input type="text" name="q" placeholder="Search docs" />
+ <input type="hidden" name="check_keywords" value="yes" />
+ <input type="hidden" name="area" value="default" />
+ </form>
+</div>
+
+
+ </div>
+
+ <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+
+
+
+ <ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="project.html">Project</a></li>
+<li class="toctree-l1"><a class="reference internal" href="license.html">License</a></li>
+<li class="toctree-l1"><a class="reference internal" href="start.html">Quick Start</a></li>
+<li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="tutorial.html">Tutorial</a></li>
+<li class="toctree-l1"><a class="reference internal" href="configuration.html">Configuration</a></li>
+<li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li>
+<li class="toctree-l1"><a class="reference internal" href="concepts.html">Concepts</a></li>
+<li class="toctree-l1"><a class="reference internal" href="profiling.html">Data Profiling</a></li>
+<li class="toctree-l1"><a class="reference internal" href="cli.html">Command Line Interface</a></li>
+<li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li>
+<li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">Security</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#web-authentication">Web Authentication</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#password">Password</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#ldap">LDAP</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#roll-your-own">Roll your own</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="#multi-tenancy">Multi-tenancy</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#kerberos">Kerberos</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#limitations">Limitations</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#enabling-kerberos">Enabling kerberos</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#using-kerberos-authentication">Using kerberos authentication</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#github-enterprise-ghe-authentication">GitHub Enterprise (GHE) Authentication</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#setting-up-ghe-authentication">Setting up GHE Authentication</a></li>
+</ul>
+</li>
+</ul>
+</li>
+<li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li>
+<li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li>
+</ul>
+
+
+
+ </div>
+ </div>
+ </nav>
+
+ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+
+ <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+ <a href="index.html">Airflow</a>
+ </nav>
+
+
+
+ <div class="wy-nav-content">
+ <div class="rst-content">
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+ <ul class="wy-breadcrumbs">
+ <li><a href="index.html">Docs</a> »</li>
+
+ <li>Security</li>
+ <li class="wy-breadcrumbs-aside">
+
+
+ <a href="_sources/security.txt" rel="nofollow"> View page source</a>
+
+
+ </li>
+ </ul>
+ <hr/>
+</div>
+ <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+ <div itemprop="articleBody">
+
+ <div class="section" id="security">
+<h1>Security<a class="headerlink" href="#security" title="Permalink to this headline">�</a></h1>
+<div class="section" id="web-authentication">
+<h2>Web Authentication<a class="headerlink" href="#web-authentication" title="Permalink to this headline">�</a></h2>
+<p>By default, all gates are opened. An easy way to restrict access
+to the web application is to do it at the network level, or by using
+SSH tunnels.</p>
+<p>It is however possible to switch on authentication by either using one of the supplied
+backends or create your own.</p>
+<div class="section" id="password">
+<h3>Password<a class="headerlink" href="#password" title="Permalink to this headline">�</a></h3>
+<p>One of the simplest mechanisms for authentication is requiring users to specify a password before logging in.
+Password authentication requires the used of the <code class="docutils literal"><span class="pre">password</span></code> subpackage in your requirements file. Password hashing
+uses bcrypt before storing passwords.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
+<span class="nv">authenticate</span> <span class="o">=</span> True
+<span class="nv">auth_backend</span> <span class="o">=</span> airflow.contrib.auth.backends.password_auth
+</pre></div>
+</div>
+<p>When password auth is enabled, an initial user credential will need to be created before anyone can login. An initial
+user was not created in the migrations for this authenication backend to prevent default Airflow installations from
+attack. Creating a new user has to be done via a Python REPL on the same machine Airflow is installed.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># navigate to the airflow installation directory</span>
+$ <span class="nb">cd</span> ~/airflow
+$ python
+Python 2.7.9 <span class="o">(</span>default, Feb <span class="m">10</span> 2015, 03:28:08<span class="o">)</span>
+Type <span class="s2">"help"</span>, <span class="s2">"copyright"</span>, <span class="s2">"credits"</span> or <span class="s2">"license"</span> <span class="k">for</span> more information.
+>>> import airflow
+>>> from airflow import models, settings
+>>> from airflow.contrib.auth.backends.password_auth import PasswordUser
+>>> <span class="nv">user</span> <span class="o">=</span> PasswordUser<span class="o">(</span>models.User<span class="o">())</span>
+>>> user.username <span class="o">=</span> <span class="s1">'new_user_name'</span>
+>>> user.email <span class="o">=</span> <span class="s1">'new_user_email@example.com'</span>
+>>> user.password <span class="o">=</span> <span class="s1">'set_the_password'</span>
+>>> <span class="nv">session</span> <span class="o">=</span> settings.Session<span class="o">()</span>
+>>> session.add<span class="o">(</span>user<span class="o">)</span>
+>>> session.commit<span class="o">()</span>
+>>> session.close<span class="o">()</span>
+>>> exit<span class="o">()</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="ldap">
+<h3>LDAP<a class="headerlink" href="#ldap" title="Permalink to this headline">�</a></h3>
+<p>To turn on LDAP authentication configure your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> as follows. Please note that the example uses
+an encrypted connection to the ldap server as you probably do not want passwords be readable on the network level.
+It is however possible to configure without encryption if you really want to.</p>
+<p>Additionally, if you are using Active Directory, and are not explicitly specifying an OU that your users are in,
+you will need to change <code class="docutils literal"><span class="pre">search_scope</span></code> to “SUBTREE”.</p>
+<p>Valid search_scope options can be found in the <a class="reference external" href="http://ldap3.readthedocs.org/searches.html?highlight=search_scope">ldap3 Documentation</a></p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
+<span class="nv">authenticate</span> <span class="o">=</span> True
+<span class="nv">auth_backend</span> <span class="o">=</span> airflow.contrib.auth.backends.ldap_auth
+
+<span class="o">[</span>ldap<span class="o">]</span>
+<span class="nv">uri</span> <span class="o">=</span> ldaps://<your.ldap.server>:<port>
+<span class="nv">user_filter</span> <span class="o">=</span> <span class="nv">objectClass</span><span class="o">=</span>*
+<span class="nv">user_name_attr</span> <span class="o">=</span> uid <span class="c1"># in case of Active Directory you would use sAMAccountName</span>
+<span class="nv">superuser_filter</span> <span class="o">=</span> <span class="nv">memberOf</span><span class="o">=</span><span class="nv">CN</span><span class="o">=</span>airflow-super-users,OU<span class="o">=</span>Groups,OU<span class="o">=</span>RWC,OU<span class="o">=</span>US,OU<span class="o">=</span>NORAM,DC<span class="o">=</span>example,DC<span class="o">=</span>com
+<span class="nv">data_profiler_filter</span> <span class="o">=</span> <span class="nv">memberOf</span><span class="o">=</span><span class="nv">CN</span><span class="o">=</span>airflow-data-profilers,OU<span class="o">=</span>Groups,OU<span class="o">=</span>RWC,OU<span class="o">=</span>US,OU<span class="o">=</span>NORAM,DC<span class="o">=</span>example,DC<span class="o">=</span>com
+<span class="nv">bind_user</span> <span class="o">=</span> <span class="nv">cn</span><span class="o">=</span>Manager,dc<span class="o">=</span>example,dc<span class="o">=</span>com
+<span class="nv">bind_password</span> <span class="o">=</span> insecure
+<span class="nv">basedn</span> <span class="o">=</span> <span class="nv">dc</span><span class="o">=</span>example,dc<span class="o">=</span>com
+<span class="nv">cacert</span> <span class="o">=</span> /etc/ca/ldap_ca.crt
+<span class="nv">search_scope</span> <span class="o">=</span> LEVEL <span class="c1"># Set this to SUBTREE if using Active Directory, and not specifying an Organizational Unit</span>
+</pre></div>
+</div>
+<p>The superuser_filter and data_profiler_filter are optional. If defined, these configurations allow you to specify LDAP groups that users must belong to in order to have superuser (admin) and data-profiler permissions. If undefined, all users will be superusers and data profilers.</p>
+</div>
+<div class="section" id="roll-your-own">
+<h3>Roll your own<a class="headerlink" href="#roll-your-own" title="Permalink to this headline">�</a></h3>
+<p>Airflow uses <code class="docutils literal"><span class="pre">flask_login</span></code> and
+exposes a set of hooks in the <code class="docutils literal"><span class="pre">airflow.default_login</span></code> module. You can
+alter the content and make it part of the <code class="docutils literal"><span class="pre">PYTHONPATH</span></code> and configure it as a backend in <code class="docutils literal"><span class="pre">airflow.cfg`</span></code>.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
+<span class="nv">authenticate</span> <span class="o">=</span> True
+<span class="nv">auth_backend</span> <span class="o">=</span> mypackage.auth
+</pre></div>
+</div>
+</div>
+</div>
+<div class="section" id="multi-tenancy">
+<h2>Multi-tenancy<a class="headerlink" href="#multi-tenancy" title="Permalink to this headline">�</a></h2>
+<p>You can filter the list of dags in webserver by owner name, when authentication
+is turned on, by setting webserver.filter_by_owner as true in your <code class="docutils literal"><span class="pre">airflow.cfg</span></code>
+With this, when a user authenticates and logs into webserver, it will see only the dags
+which it is owner of. A super_user, will be able to see all the dags although.
+This makes the web UI a multi-tenant UI, where a user will only be able to see dags
+created by itself.</p>
+</div>
+<div class="section" id="kerberos">
+<h2>Kerberos<a class="headerlink" href="#kerberos" title="Permalink to this headline">�</a></h2>
+<p>Airflow has initial support for Kerberos. This means that airflow can renew kerberos
+tickets for itself and store it in the ticket cache. The hooks and dags can make use of ticket
+to authenticate against kerberized services.</p>
+<div class="section" id="limitations">
+<h3>Limitations<a class="headerlink" href="#limitations" title="Permalink to this headline">�</a></h3>
+<p>Please note that at this time not all hooks have been adjusted to make use of this functionality yet.
+Also it does not integrate kerberos into the web interface and you will have to rely on network
+level security for now to make sure your service remains secure.</p>
+<p>Celery integration has not been tried and tested yet. However if you generate a key tab for every host
+and launch a ticket renewer next to every worker it will most likely work.</p>
+</div>
+<div class="section" id="enabling-kerberos">
+<h3>Enabling kerberos<a class="headerlink" href="#enabling-kerberos" title="Permalink to this headline">�</a></h3>
+<p>#### Airflow</p>
+<p>To enable kerberos you will need to generate a (service) key tab.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># in the kadmin.local or kadmin shell, create the airflow principal</span>
+kadmin: addprinc -randkey airflow/fully.qualified.domain.name@YOUR-REALM.COM
+
+<span class="c1"># Create the airflow keytab file that will contain the airflow principal</span>
+kadmin: xst -norandkey -k airflow.keytab airflow/fully.qualified.domain.name
+</pre></div>
+</div>
+<p>Now store this file in a location where the airflow user can read it (chmod 600). And then add the following to
+your <code class="docutils literal"><span class="pre">airflow.cfg</span></code></p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>core<span class="o">]</span>
+<span class="nv">security</span> <span class="o">=</span> kerberos
+
+<span class="o">[</span>kerberos<span class="o">]</span>
+<span class="nv">keytab</span> <span class="o">=</span> /etc/airflow/airflow.keytab
+<span class="nv">reinit_frequency</span> <span class="o">=</span> 3600
+<span class="nv">principal</span> <span class="o">=</span> airflow
+</pre></div>
+</div>
+<p>Launch the ticket renewer by</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># run ticket renewer</span>
+airflow kerberos
+</pre></div>
+</div>
+<p>#### Hadoop</p>
+<p>If want to use impersonation this needs to be enabled in <code class="docutils literal"><span class="pre">core-site.xml</span></code> of your hadoop config.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><property>
+ <name>hadoop.proxyuser.airflow.groups</name>
+ <value>*</value>
+</property>
+
+<property>
+ <name>hadoop.proxyuser.airflow.users</name>
+ <value>*</value>
+</property>
+
+<property>
+ <name>hadoop.proxyuser.airflow.hosts</name>
+ <value>*</value>
+</property>
+</pre></div>
+</div>
+<p>Of course if you need to tighten your security replace the asterisk with something more appropriate.</p>
+</div>
+<div class="section" id="using-kerberos-authentication">
+<h3>Using kerberos authentication<a class="headerlink" href="#using-kerberos-authentication" title="Permalink to this headline">�</a></h3>
+<p>The hive hook has been updated to take advantage of kerberos authentication. To allow your DAGs to use it simply
+update the connection details with, for example:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">{</span> <span class="s2">"use_beeline"</span>: true, <span class="s2">"principal"</span>: <span class="s2">"hive/_HOST@EXAMPLE.COM"</span><span class="o">}</span>
+</pre></div>
+</div>
+<p>Adjust the principal to your settings. The _HOST part will be replaced by the fully qualified domain name of
+the server.</p>
+<p>You can specify if you would like to use the dag owner as the user for the connection or the user specified in the login
+section of the connection. For the login user specify the following as extra:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">{</span> <span class="s2">"use_beeline"</span>: true, <span class="s2">"principal"</span>: <span class="s2">"hive/_HOST@EXAMPLE.COM"</span>, <span class="s2">"proxy_user"</span>: <span class="s2">"login"</span><span class="o">}</span>
+</pre></div>
+</div>
+<p>For the DAG owner use:</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">{</span> <span class="s2">"use_beeline"</span>: true, <span class="s2">"principal"</span>: <span class="s2">"hive/_HOST@EXAMPLE.COM"</span>, <span class="s2">"proxy_user"</span>: <span class="s2">"owner"</span><span class="o">}</span>
+</pre></div>
+</div>
+<p>and in your DAG, when initializing the HiveOperator, specify</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="nv">run_as_owner</span><span class="o">=</span>True
+</pre></div>
+</div>
+</div>
+<div class="section" id="github-enterprise-ghe-authentication">
+<h3>GitHub Enterprise (GHE) Authentication<a class="headerlink" href="#github-enterprise-ghe-authentication" title="Permalink to this headline">�</a></h3>
+<p>The GitHub Enterprise authentication backend can be used to authenticate users
+against an installation of GitHub Enterprise using OAuth2. You can optionally
+specify a team whitelist (composed of slug cased team names) to restrict login
+to only members of those teams.</p>
+<p><em>NOTE</em> If you do not specify a team whitelist, anyone with a valid account on
+your GHE installation will be able to login to Airflow.</p>
+<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>webserver<span class="o">]</span>
+<span class="nv">authenticate</span> <span class="o">=</span> True
+<span class="nv">auth_backend</span> <span class="o">=</span> airflow.contrib.auth.backends.github_enterprise_auth
+
+<span class="o">[</span>github_enterprise<span class="o">]</span>
+<span class="nv">host</span> <span class="o">=</span> github.example.com
+<span class="nv">client_id</span> <span class="o">=</span> oauth_key_from_github_enterprise
+<span class="nv">client_secret</span> <span class="o">=</span> oauth_secret_from_github_enterprise
+<span class="nv">oauth_callback_route</span> <span class="o">=</span> /example/ghe_oauth/callback
+<span class="nv">allowed_teams</span> <span class="o">=</span> example_team_1, example_team_2
+</pre></div>
+</div>
+</div>
+<div class="section" id="setting-up-ghe-authentication">
+<h3>Setting up GHE Authentication<a class="headerlink" href="#setting-up-ghe-authentication" title="Permalink to this headline">�</a></h3>
+<p>An application must be setup in GHE before you can use the GHE authentication
+backend. In order to setup an application:</p>
+<ol class="arabic simple">
+<li>Navigate to your GHE profile</li>
+<li>Select ‘Applications’ from the left hand nav</li>
+<li>Select the ‘Developer Applications’ tab</li>
+<li>Click ‘Register new application’</li>
+<li>Fill in the required information (the ‘Authorization callback URL’ must be fully qualifed e.g. <a class="reference external" href="http://airflow.example.com/example/ghe_oauth/callback">http://airflow.example.com/example/ghe_oauth/callback</a>)</li>
+<li>Click ‘Register application’</li>
+<li>Copy ‘Client ID’, ‘Client Secret’, and your callback route to your airflow.cfg according to the above example</li>
+</ol>
+</div>
+</div>
+</div>
+
+
+ </div>
+ </div>
+ <footer>
+
+ <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+
+ <a href="faq.html" class="btn btn-neutral float-right" title="FAQ" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
+
+
+ <a href="plugins.html" class="btn btn-neutral" title="Plugins" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
+
+ </div>
+
+
+ <hr/>
+
+ <div role="contentinfo">
+ <p>
+ © Copyright 2014, Maxime Beauchemin, Airbnb.
+
+ </p>
+ </div>
+ Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>.
+
+</footer>
+
+ </div>
+ </div>
+
+ </section>
+
+ </div>
+
+
+
+
+
+ <script type="text/javascript">
+ var DOCUMENTATION_OPTIONS = {
+ URL_ROOT:'./',
+ VERSION:'',
+ COLLAPSE_INDEX:false,
+ FILE_SUFFIX:'.html',
+ HAS_SOURCE: true
+ };
+ </script>
+ <script type="text/javascript" src="_static/jquery.js"></script>
+ <script type="text/javascript" src="_static/underscore.js"></script>
+ <script type="text/javascript" src="_static/doctools.js"></script>
+
+
+
+
+
+ <script type="text/javascript" src="_static/js/theme.js"></script>
+
+
+
+
+ <script type="text/javascript">
+ jQuery(function () {
+ SphinxRtdTheme.StickyNav.enable();
+ });
+ </script>
+
+
+</body>
+</html>
\ No newline at end of file