You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by ka...@apache.org on 2018/08/29 20:24:38 UTC
[10/48] incubator-airflow-site git commit: 1.10.0 with Updated Api
Reference
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/7d4d7628/code.html
----------------------------------------------------------------------
diff --git a/code.html b/code.html
index 02396e7..4208c5a 100644
--- a/code.html
+++ b/code.html
@@ -24,26 +24,18 @@
-
-
- <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
-
-
-
-
-
- <link rel="index" title="Index"
- href="genindex.html"/>
- <link rel="search" title="Search" href="search.html"/>
- <link rel="top" title="Airflow Documentation" href="index.html"/>
- <link rel="prev" title="FAQ" href="faq.html"/>
+ <link rel="stylesheet" href="_static/css/theme.css" type="text/css" />
+ <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
+ <link rel="index" title="Index" href="genindex.html" />
+ <link rel="search" title="Search" href="search.html" />
+ <link rel="prev" title="FAQ" href="faq.html" />
<script src="_static/js/modernizr.min.js"></script>
</head>
-<body class="wy-body-for-nav" role="document">
+<body class="wy-body-for-nav">
<div class="wy-grid-for-nav">
@@ -147,7 +139,7 @@
<section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
- <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
+ <nav class="wy-nav-top" aria-label="top navigation">
<i data-toggle="wy-nav-top" class="fa fa-bars"></i>
<a href="index.html">Airflow</a>
@@ -155,9 +147,10 @@
</nav>
-
<div class="wy-nav-content">
+
<div class="rst-content">
+
@@ -202,9 +195,9 @@
<div class="section" id="api-reference">
<h1>API Reference<a class="headerlink" href="#api-reference" title="Permalink to this headline">¶</a></h1>
<div class="section" id="operators">
-<h2>Operators<a class="headerlink" href="#operators" title="Permalink to this headline">¶</a></h2>
+<span id="api-reference-operators"></span><h2>Operators<a class="headerlink" href="#operators" title="Permalink to this headline">¶</a></h2>
<p>Operators allow for generation of certain types of tasks that become nodes in
-the DAG when instantiated. All operators derive from <code class="docutils literal"><span class="pre">BaseOperator</span></code> and
+the DAG when instantiated. All operators derive from <code class="docutils literal notranslate"><span class="pre">BaseOperator</span></code> and
inherit many attributes and methods that way. Refer to the <a class="reference internal" href="#baseoperator">BaseOperator</a>
documentation for more details.</p>
<p>There are 3 main types of operators:</p>
@@ -215,20 +208,20 @@ perform an action</li>
<li><strong>Sensors</strong> are a certain type of operator that will keep running until a
certain criterion is met. Examples include a specific file landing in HDFS or
S3, a partition appearing in Hive, or a specific time of the day. Sensors
-are derived from <code class="docutils literal"><span class="pre">BaseSensorOperator</span></code> and run a poke
-method at a specified <code class="docutils literal"><span class="pre">poke_interval</span></code> until it returns <code class="docutils literal"><span class="pre">True</span></code>.</li>
+are derived from <code class="docutils literal notranslate"><span class="pre">BaseSensorOperator</span></code> and run a poke
+method at a specified <code class="docutils literal notranslate"><span class="pre">poke_interval</span></code> until it returns <code class="docutils literal notranslate"><span class="pre">True</span></code>.</li>
</ul>
<div class="section" id="baseoperator">
<h3>BaseOperator<a class="headerlink" href="#baseoperator" title="Permalink to this headline">¶</a></h3>
-<p>All operators are derived from <code class="docutils literal"><span class="pre">BaseOperator</span></code> and acquire much
+<p>All operators are derived from <code class="docutils literal notranslate"><span class="pre">BaseOperator</span></code> and acquire much
functionality through inheritance. Since this is the core of the engine,
-it’s worth taking the time to understand the parameters of <code class="docutils literal"><span class="pre">BaseOperator</span></code>
+it’s worth taking the time to understand the parameters of <code class="docutils literal notranslate"><span class="pre">BaseOperator</span></code>
to understand the primitive features that can be leveraged in your
DAGs.</p>
<dl class="class">
<dt id="airflow.models.BaseOperator">
<em class="property">class </em><code class="descclassname">airflow.models.</code><code class="descname">BaseOperator</code><span class="sig-paren">(</span><em>task_id</em>, <em>owner='Airflow'</em>, <em>email=None</em>, <em>email_on_retry=True</em>, <em>email_on_failure=True</em>, <em>retries=0</em>, <em>retry_delay=datetime.timedelta(0</em>, <em>300)</em>, <em>retry_exponential_backoff=False</em>, <em>max_retry_delay=None</em>, <em>start_date=None</em>, <em>end_date=None</em>, <em>schedule_interval=None</em>, <em>depends_on_past=False</em>, <em>wait_for_downstream=False</em>, <em>dag=None</em>, <em>params=None</em>, <em>default_args=None</em>, <em>adhoc=False</em>, <em>priority_weight=1</em>, <em>weight_rule=u'downstream'</em>, <em>queue='default'</em>, <em>pool=None</em>, <em>sla=None</em>, <em>execution_timeout=None</em>, <em>on_failure_callback=None</em>, <em>on_success_callback=None</em>, <em>on_retry_callback=None</em>, <em>trigger_rule=u'all_success'</em>, <em>resources=None
</em>, <em>run_as_user=None</em>, <em>task_concurrency=None</em>, <em>executor_config=None</em>, <em>inlets=None</em>, <em>outlets=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/models.html#BaseOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.models.BaseOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <code class="xref py py-class docutils literal"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
+<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.utils.log.logging_mixin.LoggingMixin</span></code></p>
<p>Abstract base class for all operators. Since operators create objects that
become nodes in the dag, BaseOperator contains many recursive methods for
dag crawling behavior. To derive this class, you are expected to override
@@ -258,21 +251,21 @@ failing the task</li>
retries by using exponential backoff algorithm on retry delay (delay
will be converted into seconds)</li>
<li><strong>max_retry_delay</strong> (<em>timedelta</em>) – maximum delay interval between retries</li>
-<li><strong>start_date</strong> (<em>datetime</em>) – The <code class="docutils literal"><span class="pre">start_date</span></code> for the task, determines
-the <code class="docutils literal"><span class="pre">execution_date</span></code> for the first task instance. The best practice
+<li><strong>start_date</strong> (<em>datetime</em>) – The <code class="docutils literal notranslate"><span class="pre">start_date</span></code> for the task, determines
+the <code class="docutils literal notranslate"><span class="pre">execution_date</span></code> for the first task instance. The best practice
is to have the start_date rounded
-to your DAG’s <code class="docutils literal"><span class="pre">schedule_interval</span></code>. Daily jobs have their start_date
+to your DAG’s <code class="docutils literal notranslate"><span class="pre">schedule_interval</span></code>. Daily jobs have their start_date
some day at 00:00:00, hourly jobs have their start_date at 00:00
of a specific hour. Note that Airflow simply looks at the latest
-<code class="docutils literal"><span class="pre">execution_date</span></code> and adds the <code class="docutils literal"><span class="pre">schedule_interval</span></code> to determine
-the next <code class="docutils literal"><span class="pre">execution_date</span></code>. It is also very important
+<code class="docutils literal notranslate"><span class="pre">execution_date</span></code> and adds the <code class="docutils literal notranslate"><span class="pre">schedule_interval</span></code> to determine
+the next <code class="docutils literal notranslate"><span class="pre">execution_date</span></code>. It is also very important
to note that different tasks’ dependencies
need to line up in time. If task A depends on task B and their
start_date are offset in a way that their execution_date don’t line
up, A’s dependencies will never be met. If you are looking to delay
a task, for example running a daily task at 2AM, look into the
-<code class="docutils literal"><span class="pre">TimeSensor</span></code> and <code class="docutils literal"><span class="pre">TimeDeltaSensor</span></code>. We advise against using
-dynamic <code class="docutils literal"><span class="pre">start_date</span></code> and recommend using fixed ones. Read the
+<code class="docutils literal notranslate"><span class="pre">TimeSensor</span></code> and <code class="docutils literal notranslate"><span class="pre">TimeDeltaSensor</span></code>. We advise against using
+dynamic <code class="docutils literal notranslate"><span class="pre">start_date</span></code> and recommend using fixed ones. Read the
FAQ entry about start_date for more information.</li>
<li><strong>end_date</strong> (<em>datetime</em>) – if specified, the scheduler won’t go beyond this date</li>
<li><strong>depends_on_past</strong> (<em>bool</em>) – when set to true, task instances will run
@@ -293,32 +286,32 @@ This allows the executor to trigger higher priority tasks before
others when things get backed up.</li>
<li><strong>weight_rule</strong> (<em>str</em>) – weighting method used for the effective total
priority weight of the task. Options are:
-<code class="docutils literal"><span class="pre">{</span> <span class="pre">downstream</span> <span class="pre">|</span> <span class="pre">upstream</span> <span class="pre">|</span> <span class="pre">absolute</span> <span class="pre">}</span></code> default is <code class="docutils literal"><span class="pre">downstream</span></code>
-When set to <code class="docutils literal"><span class="pre">downstream</span></code> the effective weight of the task is the
+<code class="docutils literal notranslate"><span class="pre">{</span> <span class="pre">downstream</span> <span class="pre">|</span> <span class="pre">upstream</span> <span class="pre">|</span> <span class="pre">absolute</span> <span class="pre">}</span></code> default is <code class="docutils literal notranslate"><span class="pre">downstream</span></code>
+When set to <code class="docutils literal notranslate"><span class="pre">downstream</span></code> the effective weight of the task is the
aggregate sum of all downstream descendants. As a result, upstream
tasks will have higher weight and will be scheduled more aggressively
when using positive weight values. This is useful when you have
multiple dag run instances and desire to have all upstream tasks to
complete for all runs before each dag can continue processing
-downstream tasks. When set to <code class="docutils literal"><span class="pre">upstream</span></code> the effective weight is the
+downstream tasks. When set to <code class="docutils literal notranslate"><span class="pre">upstream</span></code> the effective weight is the
aggregate sum of all upstream ancestors. This is the opposite where
downtream tasks have higher weight and will be scheduled more
aggressively when using positive weight values. This is useful when you
have multiple dag run instances and prefer to have each dag complete
before starting upstream tasks of other dags. When set to
-<code class="docutils literal"><span class="pre">absolute</span></code>, the effective weight is the exact <code class="docutils literal"><span class="pre">priority_weight</span></code>
+<code class="docutils literal notranslate"><span class="pre">absolute</span></code>, the effective weight is the exact <code class="docutils literal notranslate"><span class="pre">priority_weight</span></code>
specified without additional weighting. You may want to do this when
you know exactly what priority weight each task should have.
-Additionally, when set to <code class="docutils literal"><span class="pre">absolute</span></code>, there is bonus effect of
+Additionally, when set to <code class="docutils literal notranslate"><span class="pre">absolute</span></code>, there is bonus effect of
significantly speeding up the task creation process as for very large
DAGS. Options can be set as string or using the constants defined in
-the static class <code class="docutils literal"><span class="pre">airflow.utils.WeightRule</span></code></li>
+the static class <code class="docutils literal notranslate"><span class="pre">airflow.utils.WeightRule</span></code></li>
<li><strong>pool</strong> (<em>str</em>) – the slot pool this task should run in, slot pools are a
way to limit concurrency for certain tasks</li>
<li><strong>sla</strong> (<em>datetime.timedelta</em>) – time by which the job is expected to succeed. Note that
-this represents the <code class="docutils literal"><span class="pre">timedelta</span></code> after the period is closed. For
+this represents the <code class="docutils literal notranslate"><span class="pre">timedelta</span></code> after the period is closed. For
example if you set an SLA of 1 hour, the scheduler would send an email
-soon after 1:00AM on the <code class="docutils literal"><span class="pre">2016-01-02</span></code> if the <code class="docutils literal"><span class="pre">2016-01-01</span></code> instance
+soon after 1:00AM on the <code class="docutils literal notranslate"><span class="pre">2016-01-02</span></code> if the <code class="docutils literal notranslate"><span class="pre">2016-01-01</span></code> instance
has not succeeded yet.
The scheduler pays special attention for jobs with an SLA and
sends alert
@@ -333,17 +326,17 @@ of this task fails. a context dictionary is passed as a single
parameter to this function. Context contains references to related
objects to the task instance and is documented under the macros
section of the API.</li>
-<li><strong>on_retry_callback</strong> – much like the <code class="docutils literal"><span class="pre">on_failure_callback</span></code> except
+<li><strong>on_retry_callback</strong> – much like the <code class="docutils literal notranslate"><span class="pre">on_failure_callback</span></code> except
that it is executed when retries occur.</li>
-<li><strong>on_success_callback</strong> (<em>callable</em>) – much like the <code class="docutils literal"><span class="pre">on_failure_callback</span></code> except
+<li><strong>on_success_callback</strong> (<em>callable</em>) – much like the <code class="docutils literal notranslate"><span class="pre">on_failure_callback</span></code> except
that it is executed when the task succeeds.</li>
<li><strong>trigger_rule</strong> (<em>str</em>) – defines the rule by which dependencies are applied
for the task to get triggered. Options are:
-<code class="docutils literal"><span class="pre">{</span> <span class="pre">all_success</span> <span class="pre">|</span> <span class="pre">all_failed</span> <span class="pre">|</span> <span class="pre">all_done</span> <span class="pre">|</span> <span class="pre">one_success</span> <span class="pre">|</span>
+<code class="docutils literal notranslate"><span class="pre">{</span> <span class="pre">all_success</span> <span class="pre">|</span> <span class="pre">all_failed</span> <span class="pre">|</span> <span class="pre">all_done</span> <span class="pre">|</span> <span class="pre">one_success</span> <span class="pre">|</span>
<span class="pre">one_failed</span> <span class="pre">|</span> <span class="pre">dummy}</span></code>
-default is <code class="docutils literal"><span class="pre">all_success</span></code>. Options can be set as string or
+default is <code class="docutils literal notranslate"><span class="pre">all_success</span></code>. Options can be set as string or
using the constants defined in the static class
-<code class="docutils literal"><span class="pre">airflow.utils.TriggerRule</span></code></li>
+<code class="docutils literal notranslate"><span class="pre">airflow.utils.TriggerRule</span></code></li>
<li><strong>resources</strong> (<em>dict</em>) – A map of resource parameter names (the argument names of the
Resources constructor) to their values.</li>
<li><strong>run_as_user</strong> (<em>str</em>) – unix username to impersonate while running the task</li>
@@ -370,7 +363,7 @@ MyOperator(…,</p>
</table>
<dl class="method">
<dt id="airflow.models.BaseOperator.clear">
-<code class="descname">clear</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/models.html#BaseOperator.clear"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.models.BaseOperator.clear" title="Permalink to this definition">¶</a></dt>
+<code class="descname">clear</code><span class="sig-paren">(</span><em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/models.html#BaseOperator.clear"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.models.BaseOperator.clear" title="Permalink to this definition">¶</a></dt>
<dd><p>Clears the state of task instances associated with the task, following
the parameters specified.</p>
</dd></dl>
@@ -541,13 +534,13 @@ task.</p>
</div>
<div class="section" id="basesensoroperator">
<h3>BaseSensorOperator<a class="headerlink" href="#basesensoroperator" title="Permalink to this headline">¶</a></h3>
-<p>All sensors are derived from <code class="docutils literal"><span class="pre">BaseSensorOperator</span></code>. All sensors inherit
-the <code class="docutils literal"><span class="pre">timeout</span></code> and <code class="docutils literal"><span class="pre">poke_interval</span></code> on top of the <code class="docutils literal"><span class="pre">BaseOperator</span></code>
+<p>All sensors are derived from <code class="docutils literal notranslate"><span class="pre">BaseSensorOperator</span></code>. All sensors inherit
+the <code class="docutils literal notranslate"><span class="pre">timeout</span></code> and <code class="docutils literal notranslate"><span class="pre">poke_interval</span></code> on top of the <code class="docutils literal notranslate"><span class="pre">BaseOperator</span></code>
attributes.</p>
<dl class="class">
<dt id="airflow.sensors.base_sensor_operator.BaseSensorOperator">
<em class="property">class </em><code class="descclassname">airflow.sensors.base_sensor_operator.</code><code class="descname">BaseSensorOperator</code><span class="sig-paren">(</span><em>poke_interval=60</em>, <em>timeout=604800</em>, <em>soft_fail=False</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/sensors/base_sensor_operator.html#BaseSensorOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.sensors.base_sensor_operator.BaseSensorOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.SkipMixin</span></code></p>
<p>Sensor operators are derived from this class an inherit these attributes.</p>
<dl class="docutils">
<dt>Sensor operators keep executing at a time interval and succeed when</dt>
@@ -584,7 +577,7 @@ override.</p>
<dl class="class">
<dt id="airflow.operators.bash_operator.BashOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.bash_operator.</code><code class="descname">BashOperator</code><span class="sig-paren">(</span><em>bash_command</em>, <em>xcom_push=False</em>, <em>env=None</em>, <em>output_encoding='utf-8'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/bash_operator.html#BashOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.bash_operator.BashOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Execute a Bash script, command or set of commands.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
@@ -592,7 +585,7 @@ override.</p>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
<li><strong>bash_command</strong> (<em>string</em>) – The command, set of commands or reference to a
-bash script (must be ‘.sh’) to be executed.</li>
+bash script (must be ‘.sh’) to be executed. (templated)</li>
<li><strong>xcom_push</strong> (<em>bool</em>) – If xcom_push is True, the last line written to stdout
will also be pushed to an XCom when the bash command completes.</li>
<li><strong>env</strong> (<em>dict</em>) – If env is not None, it must be a mapping that defines the
@@ -616,41 +609,41 @@ which will be cleaned afterwards</p>
<dl class="class">
<dt id="airflow.operators.python_operator.BranchPythonOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.python_operator.</code><code class="descname">BranchPythonOperator</code><span class="sig-paren">(</span><em>python_callable</em>, <em>op_args=None</em>, <em>op_kwargs=None</em>, <em>provide_context=False</em>, <em>templates_dict=None</em>, <em>templates_exts=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/python_operator.html#BranchPythonOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.python_operator.BranchPythonOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <a class="reference internal" href="#airflow.operators.python_operator.PythonOperator" title="airflow.operators.python_operator.PythonOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.operators.python_operator.PythonOperator</span></code></a>, <code class="xref py py-class docutils literal"><span class="pre">airflow.models.SkipMixin</span></code></p>
+<dd><p>Bases: <a class="reference internal" href="#airflow.operators.python_operator.PythonOperator" title="airflow.operators.python_operator.PythonOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.python_operator.PythonOperator</span></code></a>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.SkipMixin</span></code></p>
<p>Allows a workflow to “branch” or follow a single path following the
execution of this task.</p>
<p>It derives the PythonOperator and expects a Python function that returns
the task_id to follow. The task_id returned should point to a task
directly downstream from {self}. All other “branches” or
-directly downstream tasks are marked with a state of <code class="docutils literal"><span class="pre">skipped</span></code> so that
-these paths can’t move forward. The <code class="docutils literal"><span class="pre">skipped</span></code> states are propageted
+directly downstream tasks are marked with a state of <code class="docutils literal notranslate"><span class="pre">skipped</span></code> so that
+these paths can’t move forward. The <code class="docutils literal notranslate"><span class="pre">skipped</span></code> states are propageted
downstream to allow for the DAG state to fill up and the DAG run’s state
to be inferred.</p>
-<p>Note that using tasks with <code class="docutils literal"><span class="pre">depends_on_past=True</span></code> downstream from
-<code class="docutils literal"><span class="pre">BranchPythonOperator</span></code> is logically unsound as <code class="docutils literal"><span class="pre">skipped</span></code> status
+<p>Note that using tasks with <code class="docutils literal notranslate"><span class="pre">depends_on_past=True</span></code> downstream from
+<code class="docutils literal notranslate"><span class="pre">BranchPythonOperator</span></code> is logically unsound as <code class="docutils literal notranslate"><span class="pre">skipped</span></code> status
will invariably lead to block tasks that depend on their past successes.
-<code class="docutils literal"><span class="pre">skipped</span></code> states propagates where all directly upstream tasks are
-<code class="docutils literal"><span class="pre">skipped</span></code>.</p>
+<code class="docutils literal notranslate"><span class="pre">skipped</span></code> states propagates where all directly upstream tasks are
+<code class="docutils literal notranslate"><span class="pre">skipped</span></code>.</p>
</dd></dl>
<dl class="class">
<dt id="airflow.operators.check_operator.CheckOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.check_operator.</code><code class="descname">CheckOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>conn_id=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/check_operator.html#CheckOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.check_operator.CheckOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
-<p>Performs checks against a db. The <code class="docutils literal"><span class="pre">CheckOperator</span></code> expects
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Performs checks against a db. The <code class="docutils literal notranslate"><span class="pre">CheckOperator</span></code> expects
a sql query that will return a single row. Each value on that
-first row is evaluated using python <code class="docutils literal"><span class="pre">bool</span></code> casting. If any of the
-values return <code class="docutils literal"><span class="pre">False</span></code> the check is failed and errors out.</p>
-<p>Note that Python bool casting evals the following as <code class="docutils literal"><span class="pre">False</span></code>:</p>
+first row is evaluated using python <code class="docutils literal notranslate"><span class="pre">bool</span></code> casting. If any of the
+values return <code class="docutils literal notranslate"><span class="pre">False</span></code> the check is failed and errors out.</p>
+<p>Note that Python bool casting evals the following as <code class="docutils literal notranslate"><span class="pre">False</span></code>:</p>
<ul class="simple">
-<li><code class="docutils literal"><span class="pre">False</span></code></li>
-<li><code class="docutils literal"><span class="pre">0</span></code></li>
-<li>Empty string (<code class="docutils literal"><span class="pre">""</span></code>)</li>
-<li>Empty list (<code class="docutils literal"><span class="pre">[]</span></code>)</li>
-<li>Empty dictionary or set (<code class="docutils literal"><span class="pre">{}</span></code>)</li>
-</ul>
-<p>Given a query like <code class="docutils literal"><span class="pre">SELECT</span> <span class="pre">COUNT(*)</span> <span class="pre">FROM</span> <span class="pre">foo</span></code>, it will fail only if
-the count <code class="docutils literal"><span class="pre">==</span> <span class="pre">0</span></code>. You can craft much more complex query that could,
+<li><code class="docutils literal notranslate"><span class="pre">False</span></code></li>
+<li><code class="docutils literal notranslate"><span class="pre">0</span></code></li>
+<li>Empty string (<code class="docutils literal notranslate"><span class="pre">""</span></code>)</li>
+<li>Empty list (<code class="docutils literal notranslate"><span class="pre">[]</span></code>)</li>
+<li>Empty dictionary or set (<code class="docutils literal notranslate"><span class="pre">{}</span></code>)</li>
+</ul>
+<p>Given a query like <code class="docutils literal notranslate"><span class="pre">SELECT</span> <span class="pre">COUNT(*)</span> <span class="pre">FROM</span> <span class="pre">foo</span></code>, it will fail only if
+the count <code class="docutils literal notranslate"><span class="pre">==</span> <span class="pre">0</span></code>. You can craft much more complex query that could,
for instance, check that the table has the same number of rows as
the source table upstream, or that the count of today’s partition is
greater than yesterday’s partition, or that a set of metrics are less
@@ -667,7 +660,69 @@ single record from an external source.</p>
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
-<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>sql</strong> (<em>string</em>) – the sql to be executed</td>
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>sql</strong> (<em>string</em>) – the sql to be executed. (templated)</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.docker_operator.DockerOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.docker_operator.</code><code class="descname">DockerOperator</code><span class="sig-paren">(</span><em>image</em>, <em>api_version=None</em>, <em>command=None</em>, <em>cpus=1.0</em>, <em>docker_url='unix://var/run/docker.sock'</em>, <em>environment=None</em>, <em>force_pull=False</em>, <em>mem_limit=None</em>, <em>network_mode=None</em>, <em>tls_ca_cert=None</em>, <em>tls_client_cert=None</em>, <em>tls_client_key=None</em>, <em>tls_hostname=None</em>, <em>tls_ssl_version=None</em>, <em>tmp_dir='/tmp/airflow'</em>, <em>user=None</em>, <em>volumes=None</em>, <em>working_dir=None</em>, <em>xcom_push=False</em>, <em>xcom_all=False</em>, <em>docker_conn_id=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/docker_operator.html#DockerOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflo
w.operators.docker_operator.DockerOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Execute a command inside a docker container.</p>
+<p>A temporary directory is created on the host and
+mounted into a container to allow storing files
+that together exceed the default disk size of 10GB in a container.
+The path to the mounted directory can be accessed
+via the environment variable <code class="docutils literal notranslate"><span class="pre">AIRFLOW_TMP_DIR</span></code>.</p>
+<p>If a login to a private registry is required prior to pulling the image, a
+Docker connection needs to be configured in Airflow and the connection ID
+be provided with the parameter <code class="docutils literal notranslate"><span class="pre">docker_conn_id</span></code>.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>image</strong> (<em>str</em>) – Docker image from which to create the container.</li>
+<li><strong>api_version</strong> (<em>str</em>) – Remote API version. Set to <code class="docutils literal notranslate"><span class="pre">auto</span></code> to automatically
+detect the server’s version.</li>
+<li><strong>command</strong> (<em>str</em><em> or </em><em>list</em>) – Command to be run in the container. (templated)</li>
+<li><strong>cpus</strong> (<em>float</em>) – Number of CPUs to assign to the container.
+This value gets multiplied with 1024. See
+<a class="reference external" href="https://docs.docker.com/engine/reference/run/#cpu-share-constraint">https://docs.docker.com/engine/reference/run/#cpu-share-constraint</a></li>
+<li><strong>docker_url</strong> (<em>str</em>) – URL of the host running the docker daemon.
+Default is unix://var/run/docker.sock</li>
+<li><strong>environment</strong> (<em>dict</em>) – Environment variables to set in the container. (templated)</li>
+<li><strong>force_pull</strong> (<em>bool</em>) – Pull the docker image on every run. Default is false.</li>
+<li><strong>mem_limit</strong> (<em>float</em><em> or </em><em>str</em>) – Maximum amount of memory the container can use.
+Either a float value, which represents the limit in bytes,
+or a string like <code class="docutils literal notranslate"><span class="pre">128m</span></code> or <code class="docutils literal notranslate"><span class="pre">1g</span></code>.</li>
+<li><strong>network_mode</strong> (<em>str</em>) – Network mode for the container.</li>
+<li><strong>tls_ca_cert</strong> (<em>str</em>) – Path to a PEM-encoded certificate authority
+to secure the docker connection.</li>
+<li><strong>tls_client_cert</strong> (<em>str</em>) – Path to the PEM-encoded certificate
+used to authenticate docker client.</li>
+<li><strong>tls_client_key</strong> (<em>str</em>) – Path to the PEM-encoded key used to authenticate docker client.</li>
+<li><strong>tls_hostname</strong> (<em>str</em><em> or </em><em>bool</em>) – Hostname to match against
+the docker server certificate or False to disable the check.</li>
+<li><strong>tls_ssl_version</strong> (<em>str</em>) – Version of SSL to use when communicating with docker daemon.</li>
+<li><strong>tmp_dir</strong> (<em>str</em>) – Mount point inside the container to
+a temporary directory created on the host by the operator.
+The path is also made available via the environment variable
+<code class="docutils literal notranslate"><span class="pre">AIRFLOW_TMP_DIR</span></code> inside the container.</li>
+<li><strong>user</strong> (<em>int</em><em> or </em><em>str</em>) – Default user inside the docker container.</li>
+<li><strong>volumes</strong> – List of volumes to mount into the container, e.g.
+<code class="docutils literal notranslate"><span class="pre">['/host/path:/container/path',</span> <span class="pre">'/host/path2:/container/path2:ro']</span></code>.</li>
+<li><strong>working_dir</strong> (<em>str</em>) – Working directory to
+set on the container (equivalent to the -w switch the docker client)</li>
+<li><strong>xcom_push</strong> (<em>bool</em>) – Does the stdout will be pushed to the next step using XCom.
+The default is False.</li>
+<li><strong>xcom_all</strong> (<em>bool</em>) – Push all the stdout or just the last line.
+The default is False (last line).</li>
+<li><strong>docker_conn_id</strong> (<em>str</em>) – ID of the Airflow connection to use</li>
+</ul>
+</td>
</tr>
</tbody>
</table>
@@ -676,28 +731,89 @@ single record from an external source.</p>
<dl class="class">
<dt id="airflow.operators.dummy_operator.DummyOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.dummy_operator.</code><code class="descname">DummyOperator</code><span class="sig-paren">(</span><em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/dummy_operator.html#DummyOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.dummy_operator.DummyOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Operator that does literally nothing. It can be used to group tasks in a
DAG.</p>
</dd></dl>
<dl class="class">
+<dt id="airflow.operators.druid_check_operator.DruidCheckOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.druid_check_operator.</code><code class="descname">DruidCheckOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>druid_broker_conn_id='druid_broker_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/druid_check_operator.html#DruidCheckOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.druid_check_operator.DruidCheckOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.operators.check_operator.CheckOperator" title="airflow.operators.check_operator.CheckOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.operators.check_operator.CheckOperator</span></code></a></p>
+<p>Performs checks against Druid. The <code class="docutils literal notranslate"><span class="pre">DruidCheckOperator</span></code> expects
+a sql query that will return a single row. Each value on that
+first row is evaluated using python <code class="docutils literal notranslate"><span class="pre">bool</span></code> casting. If any of the
+values return <code class="docutils literal notranslate"><span class="pre">False</span></code> the check is failed and errors out.</p>
+<p>Note that Python bool casting evals the following as <code class="docutils literal notranslate"><span class="pre">False</span></code>:</p>
+<ul class="simple">
+<li><code class="docutils literal notranslate"><span class="pre">False</span></code></li>
+<li><code class="docutils literal notranslate"><span class="pre">0</span></code></li>
+<li>Empty string (<code class="docutils literal notranslate"><span class="pre">""</span></code>)</li>
+<li>Empty list (<code class="docutils literal notranslate"><span class="pre">[]</span></code>)</li>
+<li>Empty dictionary or set (<code class="docutils literal notranslate"><span class="pre">{}</span></code>)</li>
+</ul>
+<p>Given a query like <code class="docutils literal notranslate"><span class="pre">SELECT</span> <span class="pre">COUNT(*)</span> <span class="pre">FROM</span> <span class="pre">foo</span></code>, it will fail only if
+the count <code class="docutils literal notranslate"><span class="pre">==</span> <span class="pre">0</span></code>. You can craft much more complex query that could,
+for instance, check that the table has the same number of rows as
+the source table upstream, or that the count of today’s partition is
+greater than yesterday’s partition, or that a set of metrics are less
+than 3 standard deviation for the 7 day average.
+This operator can be used as a data quality check in your pipeline, and
+depending on where you put it in your DAG, you have the choice to
+stop the critical path, preventing from
+publishing dubious data, or on the side and receive email alterts
+without stopping the progress of the DAG.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>sql</strong> (<em>string</em>) – the sql to be executed</li>
+<li><strong>druid_broker_conn_id</strong> (<em>string</em>) – reference to the druid broker</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+<dl class="method">
+<dt id="airflow.operators.druid_check_operator.DruidCheckOperator.get_db_hook">
+<code class="descname">get_db_hook</code><span class="sig-paren">(</span><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/druid_check_operator.html#DruidCheckOperator.get_db_hook"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.druid_check_operator.DruidCheckOperator.get_db_hook" title="Permalink to this definition">¶</a></dt>
+<dd><p>Return the druid db api hook.</p>
+</dd></dl>
+
+<dl class="method">
+<dt id="airflow.operators.druid_check_operator.DruidCheckOperator.get_first">
+<code class="descname">get_first</code><span class="sig-paren">(</span><em>sql</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/druid_check_operator.html#DruidCheckOperator.get_first"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.druid_check_operator.DruidCheckOperator.get_first" title="Permalink to this definition">¶</a></dt>
+<dd><p>Executes the druid sql to druid broker and returns the first resulting row.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><strong>sql</strong> (<em>str</em>) – the sql statement to be executed (str)</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
<dt id="airflow.operators.email_operator.EmailOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.email_operator.</code><code class="descname">EmailOperator</code><span class="sig-paren">(</span><em>to</em>, <em>subject</em>, <em>html_content</em>, <em>files=None</em>, <em>cc=None</em>, <em>bcc=None</em>, <em>mime_subtype='mixed'</em>, <em>mime_charset='us_ascii'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/email_operator.html#EmailOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.email_operator.EmailOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Sends an email.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>to</strong> (<a class="reference internal" href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a><em> or </em><em>string</em><em> (</em><em>comma</em><em> or </em><em>semicolon delimited</em><em>)</em>) – list of emails to send the email to</li>
-<li><strong>subject</strong> (<em>string</em>) – subject line for the email (templated)</li>
-<li><strong>html_content</strong> (<em>string</em>) – content of the email (templated), html markup
-is allowed</li>
-<li><strong>files</strong> (<a class="reference internal" href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a>) – file names to attach in email</li>
-<li><strong>cc</strong> (<a class="reference internal" href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a><em> or </em><em>string</em><em> (</em><em>comma</em><em> or </em><em>semicolon delimited</em><em>)</em>) – list of recipients to be added in CC field</li>
-<li><strong>bcc</strong> (<a class="reference internal" href="integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook.list"><em>list</em></a><em> or </em><em>string</em><em> (</em><em>comma</em><em> or </em><em>semicolon delimited</em><em>)</em>) – list of recipients to be added in BCC field</li>
+<li><strong>to</strong> (<em>list</em><em> or </em><em>string</em><em> (</em><em>comma</em><em> or </em><em>semicolon delimited</em><em>)</em>) – list of emails to send the email to. (templated)</li>
+<li><strong>subject</strong> (<em>string</em>) – subject line for the email. (templated)</li>
+<li><strong>html_content</strong> (<em>string</em>) – content of the email, html markup
+is allowed. (templated)</li>
+<li><strong>files</strong> (<em>list</em>) – file names to attach in email</li>
+<li><strong>cc</strong> (<em>list</em><em> or </em><em>string</em><em> (</em><em>comma</em><em> or </em><em>semicolon delimited</em><em>)</em>) – list of recipients to be added in CC field</li>
+<li><strong>bcc</strong> (<em>list</em><em> or </em><em>string</em><em> (</em><em>comma</em><em> or </em><em>semicolon delimited</em><em>)</em>) – list of recipients to be added in BCC field</li>
<li><strong>mime_subtype</strong> (<em>string</em>) – MIME sub content type</li>
<li><strong>mime_charset</strong> (<em>string</em>) – character set parameter added to the Content-Type
header.</li>
@@ -711,7 +827,7 @@ header.</li>
<dl class="class">
<dt id="airflow.operators.generic_transfer.GenericTransfer">
<em class="property">class </em><code class="descclassname">airflow.operators.generic_transfer.</code><code class="descname">GenericTransfer</code><span class="sig-paren">(</span><em>sql</em>, <em>destination_table</em>, <em>source_conn_id</em>, <em>destination_conn_id</em>, <em>preoperator=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/generic_transfer.html#GenericTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.generic_transfer.GenericTransfer" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Moves data from a connection to another, assuming that they both
provide the required methods in their respective hooks. The source hook
needs to expose a <cite>get_records</cite> method, and the destination a
@@ -722,12 +838,192 @@ needs to expose a <cite>get_records</cite> method, and the destination a
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>sql</strong> (<em>str</em>) – SQL query to execute against the source database</li>
-<li><strong>destination_table</strong> (<em>str</em>) – target table</li>
+<li><strong>sql</strong> (<em>str</em>) – SQL query to execute against the source database. (templated)</li>
+<li><strong>destination_table</strong> (<em>str</em>) – target table. (templated)</li>
<li><strong>source_conn_id</strong> (<em>str</em>) – source connection</li>
<li><strong>destination_conn_id</strong> (<em>str</em>) – source connection</li>
<li><strong>preoperator</strong> (<em>str</em><em> or </em><em>list of str</em>) – sql statement or list of statements to be
-executed prior to loading the data</li>
+executed prior to loading the data. (templated)</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.hive_to_druid.HiveToDruidTransfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.hive_to_druid.</code><code class="descname">HiveToDruidTransfer</code><span class="sig-paren">(</span><em>sql</em>, <em>druid_datasource</em>, <em>ts_dim</em>, <em>metric_spec=None</em>, <em>hive_cli_conn_id='hive_cli_default'</em>, <em>druid_ingest_conn_id='druid_ingest_default'</em>, <em>metastore_conn_id='metastore_default'</em>, <em>hadoop_dependency_coordinates=None</em>, <em>intervals=None</em>, <em>num_shards=-1</em>, <em>target_partition_size=-1</em>, <em>query_granularity='NONE'</em>, <em>segment_granularity='DAY'</em>, <em>hive_tblproperties=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/hive_to_druid.html#HiveToDruidTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.hive_to_druid.HiveToDruidTransfer" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Moves data from Hive to Druid, [del]note that for now the data is loaded
+into memory before being pushed to Druid, so this operator should
+be used for smallish amount of data.[/del]</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>sql</strong> (<em>str</em>) – SQL query to execute against the Druid database. (templated)</li>
+<li><strong>druid_datasource</strong> (<em>str</em>) – the datasource you want to ingest into in druid</li>
+<li><strong>ts_dim</strong> (<em>str</em>) – the timestamp dimension</li>
+<li><strong>metric_spec</strong> (<em>list</em>) – the metrics you want to define for your data</li>
+<li><strong>hive_cli_conn_id</strong> (<em>str</em>) – the hive connection id</li>
+<li><strong>druid_ingest_conn_id</strong> (<em>str</em>) – the druid ingest connection id</li>
+<li><strong>metastore_conn_id</strong> (<em>str</em>) – the metastore connection id</li>
+<li><strong>hadoop_dependency_coordinates</strong> (<em>list of str</em>) – list of coordinates to squeeze
+int the ingest json</li>
+<li><strong>intervals</strong> (<em>list</em>) – list of time intervals that defines segments,
+this is passed as is to the json object. (templated)</li>
+<li><strong>hive_tblproperties</strong> (<em>dict</em>) – additional properties for tblproperties in
+hive for the staging table</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+<dl class="method">
+<dt id="airflow.operators.hive_to_druid.HiveToDruidTransfer.construct_ingest_query">
+<code class="descname">construct_ingest_query</code><span class="sig-paren">(</span><em>static_path</em>, <em>columns</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/hive_to_druid.html#HiveToDruidTransfer.construct_ingest_query"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.hive_to_druid.HiveToDruidTransfer.construct_ingest_query" title="Permalink to this definition">¶</a></dt>
+<dd><p>Builds an ingest query for an HDFS TSV load.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>static_path</strong> (<em>str</em>) – The path on hdfs where the data is</li>
+<li><strong>columns</strong> (<em>list</em>) – List of all the columns that are available</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.hive_to_mysql.HiveToMySqlTransfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.hive_to_mysql.</code><code class="descname">HiveToMySqlTransfer</code><span class="sig-paren">(</span><em>sql</em>, <em>mysql_table</em>, <em>hiveserver2_conn_id='hiveserver2_default'</em>, <em>mysql_conn_id='mysql_default'</em>, <em>mysql_preoperator=None</em>, <em>mysql_postoperator=None</em>, <em>bulk_load=False</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/hive_to_mysql.html#HiveToMySqlTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.hive_to_mysql.HiveToMySqlTransfer" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Moves data from Hive to MySQL, note that for now the data is loaded
+into memory before being pushed to MySQL, so this operator should
+be used for smallish amount of data.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>sql</strong> (<em>str</em>) – SQL query to execute against Hive server. (templated)</li>
+<li><strong>mysql_table</strong> (<em>str</em>) – target MySQL table, use dot notation to target a
+specific database. (templated)</li>
+<li><strong>mysql_conn_id</strong> (<em>str</em>) – source mysql connection</li>
+<li><strong>hiveserver2_conn_id</strong> (<em>str</em>) – destination hive connection</li>
+<li><strong>mysql_preoperator</strong> (<em>str</em>) – sql statement to run against mysql prior to
+import, typically use to truncate of delete in place
+of the data coming in, allowing the task to be idempotent (running
+the task twice won’t double load data). (templated)</li>
+<li><strong>mysql_postoperator</strong> (<em>str</em>) – sql statement to run against mysql after the
+import, typically used to move data from staging to
+production and issue cleanup commands. (templated)</li>
+<li><strong>bulk_load</strong> (<em>bool</em>) – flag to use bulk_load option. This loads mysql directly
+from a tab-delimited text file using the LOAD DATA LOCAL INFILE command.
+This option requires an extra connection parameter for the
+destination MySQL connection: {‘local_infile’: true}.</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.hive_to_samba_operator.Hive2SambaOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.hive_to_samba_operator.</code><code class="descname">Hive2SambaOperator</code><span class="sig-paren">(</span><em>hql</em>, <em>destination_filepath</em>, <em>samba_conn_id='samba_default'</em>, <em>hiveserver2_conn_id='hiveserver2_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/hive_to_samba_operator.html#Hive2SambaOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.hive_to_samba_operator.Hive2SambaOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Executes hql code in a specific Hive database and loads the
+results of the query as a csv to a Samba location.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>hql</strong> (<em>string</em>) – the hql to be exported. (templated)</li>
+<li><strong>hiveserver2_conn_id</strong> (<em>string</em>) – reference to the hiveserver2 service</li>
+<li><strong>samba_conn_id</strong> (<em>string</em>) – reference to the samba destination</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.hive_operator.HiveOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.hive_operator.</code><code class="descname">HiveOperator</code><span class="sig-paren">(</span><em>hql</em>, <em>hive_cli_conn_id=u'hive_cli_default'</em>, <em>schema=u'default'</em>, <em>hiveconfs=None</em>, <em>hiveconf_jinja_translate=False</em>, <em>script_begin_tag=None</em>, <em>run_as_owner=False</em>, <em>mapred_queue=None</em>, <em>mapred_queue_priority=None</em>, <em>mapred_job_name=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/hive_operator.html#HiveOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.hive_operator.HiveOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Executes hql code or hive script in a specific Hive database.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>hql</strong> (<em>string</em>) – the hql to be executed. Note that you may also use
+a relative path from the dag file of a (template) hive
+script. (templated)</li>
+<li><strong>hive_cli_conn_id</strong> (<em>string</em>) – reference to the Hive database. (templated)</li>
+<li><strong>hiveconfs</strong> (<em>dict</em>) – if defined, these key value pairs will be passed
+to hive as <code class="docutils literal notranslate"><span class="pre">-hiveconf</span> <span class="pre">"key"="value"</span></code></li>
+<li><strong>hiveconf_jinja_translate</strong> (<em>boolean</em>) – when True, hiveconf-type templating
+${var} gets translated into jinja-type templating {{ var }} and
+${hiveconf:var} gets translated into jinja-type templating {{ var }}.
+Note that you may want to use this along with the
+<code class="docutils literal notranslate"><span class="pre">DAG(user_defined_macros=myargs)</span></code> parameter. View the DAG
+object documentation for more details.</li>
+<li><strong>script_begin_tag</strong> (<em>str</em>) – If defined, the operator will get rid of the
+part of the script before the first occurrence of <cite>script_begin_tag</cite></li>
+<li><strong>mapred_queue</strong> (<em>string</em>) – queue used by the Hadoop CapacityScheduler. (templated)</li>
+<li><strong>mapred_queue_priority</strong> (<em>string</em>) – priority within CapacityScheduler queue.
+Possible settings include: VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW</li>
+<li><strong>mapred_job_name</strong> (<em>string</em>) – This name will appear in the jobtracker.
+This can make monitoring easier.</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
+<dt id="airflow.operators.hive_stats_operator.HiveStatsCollectionOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.hive_stats_operator.</code><code class="descname">HiveStatsCollectionOperator</code><span class="sig-paren">(</span><em>table</em>, <em>partition</em>, <em>extra_exprs=None</em>, <em>col_blacklist=None</em>, <em>assignment_func=None</em>, <em>metastore_conn_id='metastore_default'</em>, <em>presto_conn_id='presto_default'</em>, <em>mysql_conn_id='airflow_db'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/hive_stats_operator.html#HiveStatsCollectionOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.hive_stats_operator.HiveStatsCollectionOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Gathers partition statistics using a dynamically generated Presto
+query, inserts the stats into a MySql table with this format. Stats
+overwrite themselves if you rerun the same date/partition.</p>
+<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">CREATE</span> <span class="n">TABLE</span> <span class="n">hive_stats</span> <span class="p">(</span>
+ <span class="n">ds</span> <span class="n">VARCHAR</span><span class="p">(</span><span class="mi">16</span><span class="p">),</span>
+ <span class="n">table_name</span> <span class="n">VARCHAR</span><span class="p">(</span><span class="mi">500</span><span class="p">),</span>
+ <span class="n">metric</span> <span class="n">VARCHAR</span><span class="p">(</span><span class="mi">200</span><span class="p">),</span>
+ <span class="n">value</span> <span class="n">BIGINT</span>
+<span class="p">);</span>
+</pre></div>
+</div>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>table</strong> (<em>str</em>) – the source table, in the format <code class="docutils literal notranslate"><span class="pre">database.table_name</span></code>. (templated)</li>
+<li><strong>partition</strong> (<em>dict of {col:value}</em>) – the source partition. (templated)</li>
+<li><strong>extra_exprs</strong> (<em>dict</em>) – dict of expression to run against the table where
+keys are metric names and values are Presto compatible expressions</li>
+<li><strong>col_blacklist</strong> (<em>list</em>) – list of columns to blacklist, consider
+blacklisting blobs, large json columns, …</li>
+<li><strong>assignment_func</strong> (<em>function</em>) – a function that receives a column name and
+a type, and returns a dict of metric names and an Presto expressions.
+If None is returned, the global defaults are applied. If an
+empty dictionary is returned, no stats are computed for that
+column.</li>
</ul>
</td>
</tr>
@@ -738,7 +1034,7 @@ executed prior to loading the data</li>
<dl class="class">
<dt id="airflow.operators.check_operator.IntervalCheckOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.check_operator.</code><code class="descname">IntervalCheckOperator</code><span class="sig-paren">(</span><em>table</em>, <em>metrics_thresholds</em>, <em>date_filter_column='ds'</em>, <em>days_back=-7</em>, <em>conn_id=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/check_operator.html#IntervalCheckOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.check_operator.IntervalCheckOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Checks that the values of metrics given as SQL expressions are within
a certain tolerance of the ones from days_back before.</p>
<p>Note that this is an abstract class and get_db_hook
@@ -761,9 +1057,31 @@ against. Defaults to 7 days</li>
</dd></dl>
<dl class="class">
+<dt id="airflow.operators.jdbc_operator.JdbcOperator">
+<em class="property">class </em><code class="descclassname">airflow.operators.jdbc_operator.</code><code class="descname">JdbcOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>jdbc_conn_id='jdbc_default'</em>, <em>autocommit=False</em>, <em>parameters=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/jdbc_operator.html#JdbcOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.jdbc_operator.JdbcOperator" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Executes sql code in a database using jdbc driver.</p>
+<p>Requires jaydebeapi.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>jdbc_conn_id</strong> (<em>string</em>) – reference to a predefined database</li>
+<li><strong>sql</strong> (<em>Can receive a str representing a sql statement</em><em>,
+</em><em>a list of str</em><em> (</em><em>sql statements</em><em>)</em><em>, or </em><em>reference to a template file.
+Template reference are recognized by str ending in '.sql'</em>) – the sql code to be executed. (templated)</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
<dt id="airflow.operators.latest_only_operator.LatestOnlyOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.latest_only_operator.</code><code class="descname">LatestOnlyOperator</code><span class="sig-paren">(</span><em>task_id</em>, <em>owner='Airflow'</em>, <em>email=None</em>, <em>email_on_retry=True</em>, <em>email_on_failure=True</em>, <em>retries=0</em>, <em>retry_delay=datetime.timedelta(0</em>, <em>300)</em>, <em>retry_exponential_backoff=False</em>, <em>max_retry_delay=None</em>, <em>start_date=None</em>, <em>end_date=None</em>, <em>schedule_interval=None</em>, <em>depends_on_past=False</em>, <em>wait_for_downstream=False</em>, <em>dag=None</em>, <em>params=None</em>, <em>default_args=None</em>, <em>adhoc=False</em>, <em>priority_weight=1</em>, <em>weight_rule=u'downstream'</em>, <em>queue='default'</em>, <em>pool=None</em>, <em>sla=None</em>, <em>execution_timeout=None</em>, <em>on_failure_callback=None</em>, <em>on_success_callback=None</em>, <em>on_retry_callback=None</em>, <em>trigger_rule=u'all_suc
cess'</em>, <em>resources=None</em>, <em>run_as_user=None</em>, <em>task_concurrency=None</em>, <em>executor_config=None</em>, <em>inlets=None</em>, <em>outlets=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/latest_only_operator.html#LatestOnlyOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.latest_only_operator.LatestOnlyOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a>, <code class="xref py py-class docutils literal"><span class="pre">airflow.models.SkipMixin</span></code></p>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a>, <code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.SkipMixin</span></code></p>
<p>Allows a workflow to skip tasks that are not running during the most
recent schedule interval.</p>
<p>If the task is run outside of the latest schedule interval, all
@@ -773,7 +1091,7 @@ directly downstream tasks will be skipped.</p>
<dl class="class">
<dt id="airflow.operators.mssql_operator.MsSqlOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.mssql_operator.</code><code class="descname">MsSqlOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>mssql_conn_id='mssql_default'</em>, <em>parameters=None</em>, <em>autocommit=False</em>, <em>database=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/mssql_operator.html#MsSqlOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.mssql_operator.MsSqlOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes sql code in a specific Microsoft SQL database</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
@@ -781,7 +1099,8 @@ directly downstream tasks will be skipped.</p>
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
<li><strong>mssql_conn_id</strong> (<em>string</em>) – reference to a specific mssql database</li>
-<li><strong>sql</strong> (<em>string</em><em> or </em><em>string pointing to a template file with .sql extension</em>) – the sql code to be executed</li>
+<li><strong>sql</strong> (<em>string</em><em> or </em><em>string pointing to a template file with .sql
+extension.</em><em> (</em><em>templated</em><em>)</em>) – the sql code to be executed</li>
<li><strong>database</strong> (<em>string</em>) – name of database which overwrite defined one in connection</li>
</ul>
</td>
@@ -791,9 +1110,49 @@ directly downstream tasks will be skipped.</p>
</dd></dl>
<dl class="class">
+<dt id="airflow.operators.mssql_to_hive.MsSqlToHiveTransfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.mssql_to_hive.</code><code class="descname">MsSqlToHiveTransfer</code><span class="sig-paren">(</span><em>sql</em>, <em>hive_table</em>, <em>create=True</em>, <em>recreate=False</em>, <em>partition=None</em>, <em>delimiter=u'x01'</em>, <em>mssql_conn_id='mssql_default'</em>, <em>hive_cli_conn_id='hive_cli_default'</em>, <em>tblproperties=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/mssql_to_hive.html#MsSqlToHiveTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.mssql_to_hive.MsSqlToHiveTransfer" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Moves data from Microsoft SQL Server to Hive. The operator runs
+your query against Microsoft SQL Server, stores the file locally
+before loading it into a Hive table. If the <code class="docutils literal notranslate"><span class="pre">create</span></code> or
+<code class="docutils literal notranslate"><span class="pre">recreate</span></code> arguments are set to <code class="docutils literal notranslate"><span class="pre">True</span></code>,
+a <code class="docutils literal notranslate"><span class="pre">CREATE</span> <span class="pre">TABLE</span></code> and <code class="docutils literal notranslate"><span class="pre">DROP</span> <span class="pre">TABLE</span></code> statements are generated.
+Hive data types are inferred from the cursor’s metadata.
+Note that the table generated in Hive uses <code class="docutils literal notranslate"><span class="pre">STORED</span> <span class="pre">AS</span> <span class="pre">textfile</span></code>
+which isn’t the most efficient serialization format. If a
+large amount of data is loaded and/or if the table gets
+queried considerably, you may want to use this operator only to
+stage the data into a temporary table before loading it into its
+final destination using a <code class="docutils literal notranslate"><span class="pre">HiveOperator</span></code>.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>sql</strong> (<em>str</em>) – SQL query to execute against the Microsoft SQL Server
+database. (templated)</li>
+<li><strong>hive_table</strong> (<em>str</em>) – target Hive table, use dot notation to target a specific
+database. (templated)</li>
+<li><strong>create</strong> (<em>bool</em>) – whether to create the table if it doesn’t exist</li>
+<li><strong>recreate</strong> (<em>bool</em>) – whether to drop and recreate the table at every execution</li>
+<li><strong>partition</strong> (<em>dict</em>) – target partition as a dict of partition columns and
+values. (templated)</li>
+<li><strong>delimiter</strong> (<em>str</em>) – field delimiter in the file</li>
+<li><strong>mssql_conn_id</strong> (<em>str</em>) – source Microsoft SQL Server connection</li>
+<li><strong>hive_conn_id</strong> (<em>str</em>) – destination hive connection</li>
+<li><strong>tblproperties</strong> (<em>dict</em>) – TBLPROPERTIES of the hive table being created</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
<dt id="airflow.operators.mysql_operator.MySqlOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.mysql_operator.</code><code class="descname">MySqlOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>mysql_conn_id='mysql_default'</em>, <em>parameters=None</em>, <em>autocommit=False</em>, <em>database=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/mysql_operator.html#MySqlOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.mysql_operator.MySqlOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes sql code in a specific MySQL database</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
@@ -803,7 +1162,7 @@ directly downstream tasks will be skipped.</p>
<li><strong>mysql_conn_id</strong> (<em>string</em>) – reference to a specific mysql database</li>
<li><strong>sql</strong> (<em>Can receive a str representing a sql statement</em><em>,
</em><em>a list of str</em><em> (</em><em>sql statements</em><em>)</em><em>, or </em><em>reference to a template file.
-Template reference are recognized by str ending in '.sql'</em>) – the sql code to be executed</li>
+Template reference are recognized by str ending in '.sql'</em>) – the sql code to be executed. (templated)</li>
<li><strong>database</strong> (<em>string</em>) – name of database which overwrite defined one in connection</li>
</ul>
</td>
@@ -813,13 +1172,52 @@ Template reference are recognized by str ending in '.sql'</em>) – the sql code
</dd></dl>
<dl class="class">
+<dt id="airflow.operators.mysql_to_hive.MySqlToHiveTransfer">
+<em class="property">class </em><code class="descclassname">airflow.operators.mysql_to_hive.</code><code class="descname">MySqlToHiveTransfer</code><span class="sig-paren">(</span><em>sql</em>, <em>hive_table</em>, <em>create=True</em>, <em>recreate=False</em>, <em>partition=None</em>, <em>delimiter=u'x01'</em>, <em>mysql_conn_id='mysql_default'</em>, <em>hive_cli_conn_id='hive_cli_default'</em>, <em>tblproperties=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/mysql_to_hive.html#MySqlToHiveTransfer"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.mysql_to_hive.MySqlToHiveTransfer" title="Permalink to this definition">¶</a></dt>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<p>Moves data from MySql to Hive. The operator runs your query against
+MySQL, stores the file locally before loading it into a Hive table.
+If the <code class="docutils literal notranslate"><span class="pre">create</span></code> or <code class="docutils literal notranslate"><span class="pre">recreate</span></code> arguments are set to <code class="docutils literal notranslate"><span class="pre">True</span></code>,
+a <code class="docutils literal notranslate"><span class="pre">CREATE</span> <span class="pre">TABLE</span></code> and <code class="docutils literal notranslate"><span class="pre">DROP</span> <span class="pre">TABLE</span></code> statements are generated.
+Hive data types are inferred from the cursor’s metadata. Note that the
+table generated in Hive uses <code class="docutils literal notranslate"><span class="pre">STORED</span> <span class="pre">AS</span> <span class="pre">textfile</span></code>
+which isn’t the most efficient serialization format. If a
+large amount of data is loaded and/or if the table gets
+queried considerably, you may want to use this operator only to
+stage the data into a temporary table before loading it into its
+final destination using a <code class="docutils literal notranslate"><span class="pre">HiveOperator</span></code>.</p>
+<table class="docutils field-list" frame="void" rules="none">
+<col class="field-name" />
+<col class="field-body" />
+<tbody valign="top">
+<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
+<li><strong>sql</strong> (<em>str</em>) – SQL query to execute against the MySQL database. (templated)</li>
+<li><strong>hive_table</strong> (<em>str</em>) – target Hive table, use dot notation to target a
+specific database. (templated)</li>
+<li><strong>create</strong> (<em>bool</em>) – whether to create the table if it doesn’t exist</li>
+<li><strong>recreate</strong> (<em>bool</em>) – whether to drop and recreate the table at every
+execution</li>
+<li><strong>partition</strong> (<em>dict</em>) – target partition as a dict of partition columns
+and values. (templated)</li>
+<li><strong>delimiter</strong> (<em>str</em>) – field delimiter in the file</li>
+<li><strong>mysql_conn_id</strong> (<em>str</em>) – source mysql connection</li>
+<li><strong>hive_conn_id</strong> (<em>str</em>) – destination hive connection</li>
+<li><strong>tblproperties</strong> (<em>dict</em>) – TBLPROPERTIES of the hive table being created</li>
+</ul>
+</td>
+</tr>
+</tbody>
+</table>
+</dd></dl>
+
+<dl class="class">
<dt id="airflow.operators.oracle_operator.OracleOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.oracle_operator.</code><code class="descname">OracleOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>oracle_conn_id='oracle_default'</em>, <em>parameters=None</em>, <em>autocommit=False</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/oracle_operator.html#OracleOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.oracle_operator.OracleOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes sql code in a specific Oracle database
:param oracle_conn_id: reference to a specific Oracle database
:type oracle_conn_id: string
-:param sql: the sql code to be executed
+:param sql: the sql code to be executed. (templated)
:type sql: Can receive a str representing a sql statement,</p>
<blockquote>
<div>a list of str (sql statements), or reference to a template file.
@@ -829,19 +1227,19 @@ Template reference are recognized by str ending in ‘.sql’</div></blockquote>
<dl class="class">
<dt id="airflow.operators.pig_operator.PigOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.pig_operator.</code><code class="descname">PigOperator</code><span class="sig-paren">(</span><em>pig</em>, <em>pig_cli_conn_id='pig_cli_default'</em>, <em>pigparams_jinja_translate=False</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/pig_operator.html#PigOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.pig_operator.PigOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes pig script.</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Parameters:</th><td class="field-body"><ul class="first last simple">
-<li><strong>pig</strong> (<em>string</em>) – the pig latin script to be executed</li>
+<li><strong>pig</strong> (<em>string</em>) – the pig latin script to be executed. (templated)</li>
<li><strong>pig_cli_conn_id</strong> (<em>string</em>) – reference to the Hive database</li>
<li><strong>pigparams_jinja_translate</strong> (<em>boolean</em>) – when True, pig params-type templating
${var} gets translated into jinja-type templating {{ var }}. Note that
you may want to use this along with the
-<code class="docutils literal"><span class="pre">DAG(user_defined_macros=myargs)</span></code> parameter. View the DAG
+<code class="docutils literal notranslate"><span class="pre">DAG(user_defined_macros=myargs)</span></code> parameter. View the DAG
object documentation for more details.</li>
</ul>
</td>
@@ -853,7 +1251,7 @@ object documentation for more details.</li>
<dl class="class">
<dt id="airflow.operators.postgres_operator.PostgresOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.postgres_operator.</code><code class="descname">PostgresOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>postgres_conn_id='postgres_default'</em>, <em>autocommit=False</em>, <em>parameters=None</em>, <em>database=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/postgres_operator.html#PostgresOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.postgres_operator.PostgresOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
+<dd><p>Bases: <a class="reference internal" href="#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p>
<p>Executes sql code in a specific Postgres database</p>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
@@ -863,7 +1261,7 @@ object documentation for more details.</li>
<li><strong>postgres_conn_id</strong> (<em>string</em>) – reference to a specific postgres database</li>
<li><strong>sql</strong> (<em>Can receive a str representing a sql statement</em><em>,
</em><em>a list of str</em><em> (</em><em>sql statements</em><em>)</em><em>, or </em><em>reference to a template file.
-Template reference are recognized by str ending in '.sql'</em>) – the sql code to be executed</li>
+Template reference are recognized by str ending in '.sql'</em>) – the sql code to be executed. (templated)</li>
<li><strong>database</strong> (<em>string</em>) – name of database which overwrite defined one in connection</li>
</ul>
</td>
@@ -875,21 +1273,21 @@ Template reference are recognized by str ending in '.sql'</em>) – the sql code
<dl class="class">
<dt id="airflow.operators.presto_check_operator.PrestoCheckOperator">
<em class="property">class </em><code class="descclassname">airflow.operators.presto_check_operator.</code><code class="descname">PrestoCheckOperator</code><span class="sig-paren">(</span><em>sql</em>, <em>presto_conn_id='presto_default'</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/operators/presto_check_operator.html#PrestoCheckOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.operators.presto_check_operator.PrestoCheckOperator" title="Permalink to this definition">¶</a></dt>
-<dd><p>Bases: <a class="reference internal" href="#airflow.operators.check_operator.CheckOperator" title="airflow.operators.check_ope
<TRUNCATED>